It definitely puts it out of the range of every day users but a non-insignificant proportion of people who use it for ai have become multi-millionaires because of ai. So there's definitely no shortage of people who have no problem paying those high prices.
Which is why Apple is going to need to design around again and do something about that in house if you can design and engineer a processor or a modem you can certainly do something about memory and SSD’s, Apple buying PA Semi, Intrinsity and Anobit led to Apple Silicon, buying Infineon led to a new Apple modem the capabilities is there and the money is there in house it may take two-four years but long-term I don’t think there is any other choice.
One thing to watch for when Apple introduces the new phones coming up shortly is whether or not Apple has replaced Qualcomm in their flagship smart phones because that is coming up soon Qualcomm has given warning to their investors.
If those customers are in the market you want to develop. If they are not, money is money but if it comes from the wrong people it might slow you down.
They did. Now think they feel a stack on Minis or Studios fills the reduce needs better. The multiple machines one gets software redundancy in addition to everything else.
At this point it's basically like car models years. They rev them annually and make a few improvements but they're actually going longer between major redesigns. If they didn't bump the numbers every year people walking into stores would be worried about paying lots of money for an old phone that will soon be replaced by a much newer model.
Liquid Glass has been refined significantly in the 27 OS's and you have more control over the degree of transparency. Overall stability and performance are better as well.
It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
You should listen to the podcast Acquired, specifically Nvidia and then Jensen Huang. They basically lucked into AI. Some researcher was using Nvidia gaming cards, and reached out to them about questions on CUDA. That email eventually turned them into a trillion dollar question.
to their credit, there was a lot of work behind "luck". Jensen showed up in person in 2017 in NEURIPS and he and likely a lot of his top brass basically sat down and read the entire conference proceedings/abstracts; there was likely a lot of work behind the scenes to behind the ML research pivot.
And 2017 was _late_ in their pivot. They'd been active for much, much longer. Last winter break I sat down to watch every GTC keynote, going back to 2009[1]. Even then, he's talking about expanding to non-graphics workloads. Google's GPU paper[2] just slotted naturally into their existing narrative and were happy to support it. "fortune favors the prepared" as they say.
They definitely were sponsoring ML conferences before 2017, but symbolically, having Jensen/the CEO actually show up at a dedicated session and demonstrate detailed technical knowledge of the conference proceedings in my mind was a turning point. Albeit I remember his words to the crowd of grad students and post docs at the time: "Only Nvidia would announce their flagship card...to an audience who is completely broke!!"
What year are you talking about? When I was in grad school, around 2007, Nvidia was aggressively marketing GPUs for high performance computing. They would go to campuses, talk to professors, etc.
Yes, the whole Deep Learning thing was luck, but as with most lucky things, they ensured they were positioned to capitalize on it.
Probably cerca 2014 as that's when AlexNet was released, demonstrating that neural networks could beat traditional ML models at image recognition tasks. I recall the researchers used Cuda to optimize their training setup.
AlexNet kicked off a new wave of research around neural networks by demonstrating they could be scaled well and trained on GPUs.
I might have believe this story, if not at the same time Intel had made an expensive bet on producing not-quite-gaming cards, later looked at the same trillion dollar question.. and then almost decided that this did not bring enough luck to keep spending.
> "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy"
This is clearly a mis-statement, they have a whole annual conference for developers. Maybe they mean specifically AI devs.
I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.
I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
I was getting semi-useful results from a 128GB M4 Max. That was a few months ago, and the models have improved (quite a bit) since then, but now I'm happy to send my $20/month to get Claude code.
It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
Bloody training data full of humans demanding sleep. I tells ya...
> "not do significant work this at this late hour"
Is this Claude code? Or your local? I assume Claude? I'm more than a little staggered by this, like, it makes no sense! It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.) The LLM just... decided? I'd be so mad.
I think 24gb is the bare minimum for a local qwen3.8 based setup. I've used qwen3.6 and it's not as straightforward as "can it replace <insert the most cost-effective cloud solution today>"
Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.
Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.
The principle of KISS keeps coming to mind when I consider local computing. I'm looking forward to the day we can just run Opus-level models at 100 tok/sec on consumer hardware.
But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)
I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).
IMO local models require a substantial amount of prompt+harness engineering to get in the neighborhood of what you'd get from a cloud model. Which isn't a bad thing, you'll learn a whole lot about how these things work.
What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.
When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.
What you’re describing is exactly what I’ve experienced in my time testing out this stuff locally, and I had a hard time figuring out what exactly to blame.
I keep hoping that one day some comment is going to paste a link to some kind of idiot-proof guide or piece of software that’s “90% as good as Claude but running local.”
And by 90% I don’t mean that the model is 90% as good or runs 90% as fast, more like all the other stuff you mentioned is set up out of the box.
16GB VRAM could load a small quantised qwen 27B model but it would be a ways away from a frontier cloud model.
Though keep in mind not being beholden to shenanigans from said cloud companies (and interference from government entities!) is definitely worth something intangible.
I've been experimenting with local models on an M5 Max MBP w/ 128GB of RAM since March of this year. Generally I've had very good results. Where things were lacking initially was with tool calling and the need to rely on tool calling for functionality like web search, which is otherwise well integrated in the cloud models. There is also a lot more work required on the harness side, however at this point (August 2026) there is not only much better tool calling in local models, but community supported projects have built good harnesses. Pi.dev and OpenCode + a SOTA local model is /very/ /very/ capable these days. Using LM Studio's built-in chat with a decent system prompt and proper tuning with local models is /very/ capable these days. Cloud models are still better, but it's the "harness" (not in the desktop app, but in the backend) that makes it so for the most part.
One use case I find cost effective is using it as a voice assistant for Home Assistant. API pricing on models is very weird compared to the normal chats, so I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.
>I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
I just ordered a new Mac Studio M5 Max 128GB $5899 ($6400 with tax) to be able to run a bigger "consumer size" model in the 70B parameter range (~96 GB). That said, I have no illusions that this expensive setup with a Qwen Flash coding LLM will be comparable to a $20/month subscription. Even upgrading to an even more expensive Mac Ultra 256GB for $10000 to hold a bigger model still won't be comparable. Apple hasn't shipped my Mac yet and I'm still considering cancelling it and downgrading to a smaller 64GB RAM config ($4299) to save $1600.
Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I use some local LLM for easy tasks or when I hit rate limits. No issues with privacy so that wasn't part of the motivation at all. I just wanted some local AI capability to augment a subscription. I've not totally convinced myself of the cost/benefit of this.
Based on today's consumer hardware landscape, you're paying very high prices for crippled capability compared to the cloud AI subscriptions. We're also in a transition period where the next iteration of hardware improvements have some compelling features for local AI. Apple's upcoming M7 (2027 or 2028) is anticipated to have better GPU and neural engine to help with prefill TTFT. AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc. Maybe apply my $1600 savings towards those newer products. Those future products are still very expensive but maybe the cost/benefit will be better.
you can get 4xGB10 for <20K so that gets you about the same tg and pp will be probably better. Power consumption though will be something like 200W idle so that's a bummer. And you get VLLM and SGLANG unlike them mac where one has to use oMLX (nice but not the same concurrency or cache performance)
I have 128GB M5 Max laptop and sill prefer to run things on other boxes in basement because its no fun to have the primary device being overloaded.
> Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I could use some local LLM for the easy tasks or when I hit cloud rate limits.
The maths don't check. With Deepseek Flash one goes a very long way with 1600$ - even 10$/month, for easy jobs, are more than 13 years, and at a higher quality.
Oh no doubt. But one does have the guarantee that no bits left their home and that's a big one for some.
The low hanging fruit stuff for me is more something I use it for because I have the local LLM setup running anyway. It wasn't the reason I bought it, but now that it's there I might just as well use it as much as I can.
> Oh no doubt. But one does have the guarantee that no bits left their home and that's a big one for some.
No, this is a big misconception, and part of the cargo cult.
Use cases like the parent's are essentially about having a local LLM handle the leftover tasks. By that point, a lot of bits (main/big tasks) have already left home anyway.
It not only about it being an expensive setup (or not), and also other considerations:
- There's no guarantee of the $20/month service, and it likely has some limits compared to dedicated hardware token wise.
- Model are becoming more and more efficient, in many cases an M1 Max Mac Studio is still capable with 32 GB. 128 GB ram may not be the necessary baseline.
- Folks may think they want to only have a general model running locally (it's the comparable after all from the cloud providers), but we have to remember if the tasks we're trying to do ultimately are more specific than general and if there's space for the smaller models to do that.
> AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc
Big upgrade to memory capacity but memory speed is only going up by a few percent, so its still going to be slow with more than a few B active params (I have one)
Keep the memory. You’ll be glad you did when you realize that you’re better off with a solid coding model plus a good voice model and also a lightweight all-rounder all running at once isn’t of loading dynamically (slowly). It also helps if you want to be able to run a browser, IDE, and container environment.
I think the M5U Ultra 96gb is the sweetspot in that price range. It has more compute and bandwidth so you get to run models better sized to its hardware. I believe the Max would be too slow; personally I'm getting this SKU because I think it'd suck to get the 128gb Max and then discover it's too slow, and end up just using cloud providers anyway.
I agree about getting the Ultra if you're interested in AI (LLM) inference speed. I'm a little perplexed as to why there isn't a RAM option in between 96GB and 256GB, though. For instance, I believe Deepseek v4 flash runs a lot faster on (https://github.com/antirez/ds4) with 128GB than 96. I assume the higher SKUs have low enough sales that Apple doesn't want to be manufacturing too many different RAM configs as that would eat into their margins. (I say this as someone interested in AI hardware in general, even the 96GB is out of my price range).
It's the upsell ladder. This is how they get you to buy a more expensive tier or two. At first I was only going to look at 64gb or 128gb Max, but ended up here.
I also think it’s possible that it may be best to continue waiting on the Mac side of the house despite the increasing prices.
I think that some of the hardware design folks have been blindsided by AI demand and we haven’t really gotten that next generation AI hardware yet, to the point where buying M5 isn’t going to make sense in a couple of years.
Rumors seem to be that the M7 is the generation that Apple is looking to push AI performance much further.
I’m not sure that Apple anticipated this specific route that computer hardware has gone and I don’t think M5 and previous iterations were really specifically architected for local AI performance, more like they happened to be pretty good at it.
> Rumors seem to be that the M7 is the generation that Apple is looking to push AI performance much further.
What specifically does that mean? The M5 series has 10 cores per 128 bits of memory bus; are the cores unable to keep up with the RAM? I thought they did and memory bandwidth was usually the bottleneck. But the memory bus is already very highly clocked and goes up to 1024 bits wide so it's hard to picture memory bandwidth having a huge leap.
LLMs token generation is memory bandwidth constrained. If the M7 has double the memory bandwidth as some speculate [1], then it will help with LLM performance. There is also expectation (possibly unfounded) that the GPU will have improved matrix performance to help prompt processing as well.
14GT/s is a goal but it might be a while, I think 11-12 is more likely in that time frame.
50% more pins... we'll see. If they can reasonably make that fit then even more shame upon the traditional desktop CPU makers for sticking with 128 bits for so long.
I’ve been running DeepSeek 4 Flash, Qwen 27B and Qwen 9B on local hardware. They work well for coding and document review tasks. I think Qwen 9B local on a 5090 might be legitimately helpful for small task agents in omp, since it is ridiculously fast. But my motivation is that I have data that I unfortunately can’t share with 3rd parties.
I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.
I have a RTX PRO 6000 96GB when the pricing was way better than now i also have a RTX 5090 too.
What I noticed is that (1) the great local models are optimized run inference (diffusion & LLMs) well on 32GB VRAM <= GPU's because that that's what the target has ...
(2) The quality of local models (esp. in diffusion) is increasing faster than the need for more VRAM - additional reason for the value of these FAST GPUs to increase!
(3) RTX PRO 6000 96GB is really great for fine tunes (ai-toolkit) :) but doesn't outperform my RTX 5090 with inference by anything significant on the good local models.
I have never run an AI job on a Mac, i also have doubts about performance and compatibilities - since the reviews almost never compare directly.
In my experience so far, separately from privacy concerns there's a specific use case where cheap local shines--when unlimited shots on /goal with a dumber model is better than limited time with a smarter model. This looks less like the assisted-coding scenario that's commonly brought up as a good local scenario, because if you're searching for a fast and accurate solution to a single blocking problem, the bare minimum for a model that can do that is a 24GB dGPU or a 64GB Mac. Instead, this looks more like a Hermes agent on an Raspberry Pi driving OpenCode on an old gaming computer with just enough RAM and VRAM to handle an MoE, churning out something overnight that would quickly exhaust the subscription plans, like a knowledge graph for a large document corpus.
A huge benefit of local setups at our studio is that a lot of our software can't run headless, so when we're having agents work in Blender or in Unity etc with MCP that'd otherwise eat up our normal computer use. (Try to have two people try to work in the same Unity editor at once... then try ten!)
We also built some QA agents that are always playing our games from the same builds a player would and flagging things to fix/improve; that alone needs the game focused and front-and-center so it can properly screen-capture for deciding what inputs to take next (and for screenshots/replays), which also means we can't really do any hands-on work at all on the machine when it's running.
Having a separate (and tiny) machine for all of this has been great. We don't bother with local models because, you're right, the $20/month sub is way better than anything that can run on small consumer hardware atm.
I might be wrong but subscription models don't give you API access. I'm only interested in API access when it comes to personal use, so local models running for free makes sense for me.
That said I have an RTX 5090, not a Mac Mini, so it's not exactly the same level of performance... The latest open models run at 200 tpm at around 30B params.
You have good enough hardware to run good models comparable with Gemini and ChatGPT.
What they are doing that you might not be is context caching, retaining a user profile about you in vector db, and adhering to strict output templates - often invoking multiple different prompts to assemble the end response to the user.
You’ll notice the mainstream ones always kinda answer in the same way. They are also able to mix code and images with text (multi-modality) because they are breaking down and routing your prompt to different models then assembling a result.
That’s all just software, has nothing to do with the model.
More people imo should be experimenting with it rather than chasing better models. This area is where all the differentiation is happening in the services, and what gives an AI provider like Gemini or Vidu or Singify an edge over running the equivalent locally
Ignore these shills in the replies. Models peaked a long time ago.
Yes there are smaller ones like mistral 7b vs larger ones which deliver only a slight improvement, but tbh most of the larger ones are worse than Mistral 7b.
The major providers like Google Gemini and ChatGPT and the “me too” company Anslopic are doing context caching (fact check me) and db retrieval to build responses.
Their responses adhere to templates and multiple different models are requested (one for the personalized description, one for the code solution, one for the image diagram, etc.) this is called multi-modality. Gemini and 4o stole most of their UX ideas from a project called Ragdoll (the first ever multimodal chat AI software released months before 4o).
You can make the same exact quality of AI chat service at home setting up harnessing like I described here namely the context caching and multi-modality. That is what differentiates an experience like Gemini from running local Ollama or Stable Diffusion or whatever where you are just getting a raw response from 1 model at a time.
I have an older M2 Mac mini that does the OCR and visual description of all my screenshots. Screenshots are stored on my NAS.
I like to screenshot things as a quick way to remember. They are things that I would not be comfortable sending a cloud provider (customer data, prototype screenshots, bank dispute details).
It runs Qwen3.5:9b and glm5.2-ocr with Ollama and uses about 10GB of RAM. It automatically releases the models from RAM after 5 minutes of inactivity so it is pretty seamless to leave running in the background.
All the details are stored in a simple webapp with a SQLite db that I can search through.
> They are things that I would not be comfortable sending a cloud provider
It's also an old machine that the commenter already has; it's intellectually dishonest to compare it to the price of a brand new, 4-iteration-newer machine.
One question I keep meaning to research is what the cloud options are for private-but-hosted on higher end hardware. Maybe rent an H100 (or whatever) and host something that's just too much to fit in 96GB on my M2 max. I wonder how the costs pencil out on that.
Neither openai or xai. Anthropic maybe but not likely. Mistral is the most likely one because they're under the EU laws but I think they focus more on commercial these days.
This is such a tired argument and it seems to be parroted every single time someone talks about local models on hacker news.
Yes, of course the most economical path is to hand over all your data and become fully dependent on a cloud provider who is already operating as scale, hoping that they won't change/remove models, hamstring capabilities, or raise prices.
If this were a thread about hosting your own email or blog or cloud photos, you'd have plenty of people out here telling you how easy it is to do it yourself instead of relying on Gmail for email or WordPress/Medium/Substack for blogging, or iCloud for cloud photos.
And yet, without fail, every single thread about self hosting local models seems to have some copy/paste form of this cost-savings argument.
Where is the appreciation for this cool thing GP built? Where is the appreciation for the desire to figure out how to host your own version of the incredible capabilities that were not available merely a few years ago? And why, on this site of all places, would someone advocate trading all of the knowledge and independence gained from learning how to host something like this ourselves in favor of throwing it all over the wall to Google?
It's quite shocking to me how many experienced, tech-savvy people, who used to care about cookies and ad tracking - are now willingly sending their business strategies, highly confidential contracts, and intimate personal issues to a cloud provider because "it is only $0.0x per million tokens!".
> It's quite shocking to me how many experienced, tech-savvy people, who used to care about cookies and ad tracking - are now willingly sending their business strategies, highly confidential contracts, and intimate personal issues to a cloud provider because "it is only $0.0x per million tokens!".
Because there are more privacy guarantees there, depending on the provider. "But what if they violate their contract!" is some pretty tin-foil hat stuff.
How is this any different than a business running their website out of the cloud, assuming you are using a provider with appropriate contractual terms?
You can care about tracking and ads but still be comfortable storing your backups in the cloud, and many have been for quite awhile, even sometimes without encryption - that is totally different than e.g. Meta actively trying to track you and understand your relationship graph and your purchases etc.
I'm not sure about the tin-foil-hattedness of worrying about them violating their contract. But that's by-the-by. It is definitely not tin-foil-hat to worry about the data being taken in a breach.
Depends on how you use it. You can use AWS in a way that protects your data even if Amazon is compromised. You can even run LLM inference that way, but it doesn't seem at all common.
I would say tech workers have very weak class consciousness and understanding of power structures. Those who care about not ceeding power is a tiny minority compared to say among MDs and lawyers and their guilds/unions.
I don’t understand the willingness to give up privacy so easily, particularly if you are developing something that you plan to monetize somewhere down the road.
I’m pretty sure that all of those disclaimers that all the AI model makers have for you to sign off on to say that they’re not responsible for anything that might go wrong if your work gets copied accidentally and used someplace else wink wink?
You know the lawsuits for that particular aspect are incoming in the future…
Yeah I use it for hobby only. I would use cloud models to develop new stuff, I don't really care if it gets lost because if I were to publish it it would be FOSS anyway. I hate entrepreneurism and monetisation so I'm happiest being a salaried employee with many hobbies :)
But no way whatsoever I'm uploading my personal files, photos, emails, chats into cloud AI. No way.
I think the price is beyond that though. AFAICT, there's not competitively fast image or video generation on Mac. To buy it will cost me $4000-$12000. So I rent.
having a 64GB mac mini m4 pro the last few years with some increasingly capable usefulness has kept me interested in this stuff in a way that using a paid platform wouldn't have. Similar to running K8s in a homelab, something about interacting with the hardware makes it more engaging/interesting, for me at least.
In general, I'm a big believer in doing more with fewer resources, within reason, and think having local setups really helps me be mindful with what's happening under the hood with these systems and managing context efficiently to get high quality results.
I guess it depends on what you're trying to do. I've run a few LLMs on my 64gig Mac but anything image or video related is ridiculously slow compared to even old NVidia on my PC.
Have you been using an MTP setup? I've been having pretty good luck with the Qwen models with the built in MTP heads via https://mtplx.com at around Q4. My main driver rig is an M5 Max MBP work got for me a few weeks ago. Hitting 1000-1200 TPS prefill on Qwen 3.8-Flash-Next there.
A $20/month Gemini subscription is truly all you need, then yeah, sure.... obviously a homelab setup is a ridiculous alternative on a pure cost basis. For most people doing "real" work with LLMs 40+ hours per week, a more apt comparison would be one or multiple $200/month subscriptions. At which point the break-even point of a homelab is much sooner.
However, most people running homelabs are doing it for other reasons. Independence, learning, and/or privacy issues.
I have not spun up my own homelab, but, I have researched it and the electricity costs can be mitigated to a large extent.
One... the GPUs can be massively clocked down during idle state, to the point where the fans can be shut off as well. The machines themselves can be shut down and wait for a magic wake-on-LAN packet if needed.
Two... even under load, the GPU cores can be significantly underclocked with very little performance loss. The GPU and VRAM/HBM clocks are independent, and the GPU is largely bottlenecked on VRAM/HBM so you can just drop the GPU speed. A commonly reported figure I saw was, basically, 350W nominal cards being underclocked to consume "only" 200W under load with ~5-10% perf loss.
This all assumes you're using discrete GPUs and not AIO systems like a Mac Studio which is going to be pretty efficient just by design; they idle at 35W or so and in practice max out at a few hundred watts. I believe DGX Spark and Strix Halo are similar.
I must stress that this is second-hand anecdata here, admittedly, but my understanding is that it can be pretty manageable.
Yes but I also get a full fledged computer in the deal. I can sell it later. I can use it for all sorts of things like games and browsing and video editing. Paying for Gemini for other tasks is also in the mix but at the end of 4 years I get...nothing.
I personally use Apple Photos for this. It stores the original, plus makes them nicely searchable, so I have a Hazel action that takes screenshots from the desktop (and from my NAS where mobile devices back them up) and imports them.
For the amount of tokens you get, based on your comment, ALL subscriptions are heavily subsidized, and the most expensive ones are the most subsidized.
For OpenAI and Anthropic, the $100 subscriptions cost 5x the $20 subscriptions and give you 5x the tokens. And the $200 subscriptions are 10x the cost for 20x the tokens. (Tokens cost 50% as much.)
I have a RX 9070 also. I run llama-swap with a fill-in-middle 7B model, local 9B model, and it proxies up to OpenRouter for the bigger stuff. I think that's where the sweet spot is right now.
I've spent $2 in the last 2 weeks on OpenRouter. I've been trying to only use the medium sized models that I would otherwise be able to run on a nice local setup. That nice local setup would cost ~$4k. I don't know what the operating cost would be, but I would be concerned that my home electricity would cost more than at a datacenter. It just doesn't make sense right now except for privacy reasons.
I'm probably going to hoarde open weights models in the ~31B range until memory costs fall in a few years. Then, I'll buy some hardware to run at home just so I feel more sovereign over my stack regardless the cost/token speed.
In my limited experience, not quite yet but we are damn close. Qwen 3.8 27b is it. If I could run this as a decent speed, I would no longer need cloud models at all. I'm actually currently trying it out in the cloud to pay for the inference speed but the model is fully runnable at home.
I realistically costs $5-10k to replicate a ChatGPT like agent. And it doesn't scale.
That's still really close. And models and quantization etc keep improving.
I'm absolutely positive that I'll be switching to mostly local AI in the next 5 years.
Good 60-70 tg and 2K pp Qwen 3.8 27B FP8 can be had for about 5-6K (2xR9700 + PC)
Gives about 3-4 concurrent sessions with full 262K
Fast 150+ tg and 2-8K pp Qwen 3.8 27B nvfp4 is about 8K (5090 +PC)
Gives really only one concurrent session that flies because kv caching is not perfect for ninfer https://github.com/Neroued/ninfer
both are very serviceable, I prefer FP8 on 2xR9700
But, yes it doesn't scale that well but in 5 years the same hardware should still be very capable of running some great MoE models, for example Qwen 3.6 35BA3B on 5090 can fly at 600 tg
I could believe that especially with some of the chips playing catch-up.
E.g., the M7 chip is rumored to be the one where Apple has poured really serious effort into local AI performance where previous generations seem to have mostly been coincidentally good at it.
Maybe this is an incorrect opinion but I don’t personally think that the M1-M3 or maybe M4 or even M5 chips were designed with LLM inference in mind at all. These were designed with things like video rendering, image/video ML, and rasterization performance in mind.
I’d be curious to use them on larger data sets. Log files, for instance. High volume, might be low value per line but not much cost per token if you already have a gpu to interpret them using a LLM.
Data privacy and "unsafe" models are pretty valid reasons to use local models. If I want to generate violent images/text you cannot do that using cloud models.
From what I’ve been seeing, the Mac studios do look like they have potential. I was looking to drop $10k-$15k on one until recently. After comparing a Radeon 7900 XTX vs Ryzen Halos 128GB vs M1 MacBook Pro 64Gb, I landed on just getting an external closure setup with Nvidia RTX 5090.
The model I’m specifically targeting to use at high speeds is Qwen 3.8 27b @q4ks. This model actually proved to be good at coding (it sits somewhere between Sonnet 5 and Opus 5 capability). M1 got 10 tok/s, Ryzen Halo 20tok/s, and Radeon 7900 XTX 50tok/s (can only do 128k context window in Radeon card).
The prefill gets extremely slow around 50k tokens in context window (whatever prompt processing stage entails could be wrong about phases here). It takes about 2 hours to fill the context.
Even with a drafter model intended for speed instead of mtp, I can’t get past 70tok/s, still is extremely slow to process prompts as context grows, and drops down to 40-50tok/s anyway making this config still moot for improvement on my Radeon card.
The only thing I can point to slowing me down is bandwidth of the card itself.
I am waiting to actually get my 5090 right now and I am betting that the 1700 Gbps of capacity will fix my prompt processing speeds. I don’t need full PCIe lane bandwidth to serve my house I just need to load the full model into vRAM and let the GPU do its thing.
Additional benefit to the external enclosure route is being able to migrate the inference between devices more easily. I can develop out the infrastructure then migrate the card to be hooked up to a shared node in the house with all the tools necessary for my family to take advantage of the privacy enhancement that comes with local inference.
How are you actually using the local model? I've played with Qwen 3.8 27b on ollama and the coding harnesses (Claude Code and OpenCode) seem to fail way more often then using the cloud models. And by fail, I mean the edits don't apply cleanly, it goes to add python code, but doesn't indent it properly, or the edit doesn't apply and so it tries again and again and eventually wipes out a different function then it "intended". It just gets really frustrating compared to the relative stability of Claude Cloud.
My use-case is only coding, every model sucks at writing good literature and there is no way around that (have had people try to debate me on this, but it's a taste thing, I have extensive English writing skills from my school years).
Prior to two weeks ago, I was just using Pi and Ollama.
I have tried my hand at putting together a few harnesses and I finally landed on what I like. Been working on this small app to handle running llama-server for me from any device that has the llama-cpp stack setup: https://github.com/SamInTheShell/loom
try using pi harness, hae not encountered these sort of problem myself
also yuou can ask codex to look at the transcript and figure out the solutions to tool call failures that way
Thanks for sharing, I'm definitely trying it out after I get through my project milestones for the 5090. The README claims 700tok/s for Qwen 3.8 27b, that would be amazing, I'm only expecting an increase from my ~50tok/s on my Radeon to 200tok/s on the 5090.
Just wanted to share, I had a SaaS AI drop me a script to bench ninfer against llama-cpp and it is impressive. The place that it's doing better at than llama-cpp seems to really be late in the context window.
I was looking at $10k Mac Studio with M5 Ultra and 256 GB for local experiments, but then struggled to find what really good modern model I can fit into it. Yes, it can run a good dense 27B at Q8 with plenty of context, but what beyond that? IIUC, some Deepseek flash variants at Q4 are also feasible, but I am not sure if the quality will be good. They also don’t run that fast, like about 30 t/s
So if I stay within 35B, especially MOE, my M5 Pro 64GB MBP can also run them well, and it can do plenty of other stuff too including gaming. While 256 GB with such RAM bandwidth and powerful GPU sounds like fun on paper, it doesn’t seem to be the next level compared to 64 GB
There are many enterprise environments where running modern models is... difficult. Rather than fighting security for months, a user could petition for a mac studio and have rough cost parity with a z.ai subscription.
if they are a heavy user, perhaps they string 4x together.
Right now sweet spot is voice transcription. Meeting recording apps are genuinely better locally than in cloud. Can run on an M1 easily. Latency matters. I built https://github.com/scosman/Biscotti and see zero reason to use cloud ever again.
LLMs are harder: not much useful below 12B, and the 700B+ ones are really much better. Models like Qwen 3.8 27b show promise: in a few years pretty good local AI should be in reach for anyone willing to buy a $1000 computer (but who knows what your $20 sub buys you then).
It's not that far off anymore. On my 7900 XTX 24GB, I can run Qwen3.8 27B with 131K context at Q4_K_M (55 tok/s with MTP). Excluding hardware cost, it's about $0.02 tok/M in and $0.40 tok/M out (cached in $0.0001). On OpenRouter, that would cost more than 10x what it actually costs me.
Of course, 131k context at 4-bit quant is a trade off, but even then, it's VERY capable. It doesn't feel that far behind something like GPT 5.6 Luna.
Image and Video gen is superior locally, because you can tweak more, use LoRAs, use whatever model fine tunes, and generate uncensored content, plus as you're often cherry picking from multiple gens it ends up being cheaper for comparable quality as well.
Local coding is a step down but good enough for a lot of things if you have privacy concerns.
Wild oversimplification, and benchmarks vary widely, but I've read a lot of benchmarks suggesting that Qwen3.8-27B (xhigh effort) competes with near-frontier models at a lot of coding tasks. To the best of my understanding it's not going to run very feasibly in 16GB of VRAM at usable quants however.
r/LocalLLM and r/LocalLlama are noisy, but valuable sources of anecdata if you have the time (or the tokens, hah) to comb through them. You are going to see a lot of modest setups there, and also guys with $20K+ of hardware.
The two things (besides my bank account) that keep me from investing heavily in local are (1) we are not guaranteed to get a steady release of open models in the future (2) a lot of the "fun" stuff LLM stuff that interests me involves orchestrating lots of parallel agents, which of course multiples the hardware you need to achieve it.
For example, I've been having good results having both Sol and Opus review the same PR, and then I have them cross-review each others' PRs. A next step I'd like to consider is maybe having a swarm of Luna agents review the same PR and have them fight it out... maybe with Sol doing final arbitration? I suspect 5-10 Lunas might outperform a single Opus. Or maybe not. But at any rate, that would be impractical in a homelab without a pretty big hardware (or time) budget.
For software development tasks, Qwen 3.8 27B is genuinely excellent, but you need 32+ GB of VRAM to run it well with decent context, and enough memory bandwidth and compute to run it at a decent pace. With an M5 Max Mac Studio, you can do that decently well.
Local inference can't compete with cloud on speed, intelligence and economics. It's all about freedom, privacy, control, sovereignty.
It's about not having to accept any of the stupid "terms" of the corporations. It's about doing things the big labs don't allow you to do, like cybersecurity stuff, or even just chatting with the AI about some wrongthink.
Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.
It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.
If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.
Apple doesn't design GPUs on-par with Nvidia's efficiency yet. They need an architectural overhaul to be a serious competitor, which is what I'm expecting is queued up for M7.
Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?
> Apple doesn't design GPUs on-par with Nvidia's efficiency yet
How much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
For prefill, it's basically all that matters. Long-horizon agent tasks, session compaction, file reads and context manipulation will all hit the compute bottleneck in regular usage, incurring several minutes of latency on most Apple Silicon chips, regardless of RAM.
It's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.
Probably two generations away. I’m more interested in how much uplift/speed and more importantly what is the power usage is required for the new computers Apple is shipping particularly for the Studio versions.
Apple isn’t the company that eats margins but they are company that would design around the problem and I think that’s what they will do after all, they have the design and engineering and plenty of money because they didn’t burn it on AI models or data centers.
With AI Apple is doing their classic strategy I guess: wait and then follow with something that learns all the lessons the pioneers learned.
They're letting everyone else pay for model training, with the obvious end of the road being open weights.
They're letting everyone else overspend on first-generation AI data centers before the chip industry has truly optimized for today's AI designs (which will reduce data center footprint).
They're letting everyone else play with and pioneer UI ideas.
Meanwhile all they've done is put a few toes in the water: "Apple Intelligence" which is barely anything, and adding AI acceleration to their GPUs and doing a bit of up-market marketing to AI.
So yeah they'll probably follow with a second generation Apple Intelligence that incorporates everything everyone else pioneered that worked, and an M7 or M8 line of chips with a GPU augmented with whatever approaches the other chip companies found worked best for running models... and built around the state of the art model architecture the industry finally converges on. By then RAM will probably be cheap again, so you'll get a 14" MacBook Pro with an M8 with a tensor-GPU and 512GiB of RAM.
A Mac Mini with M5 Pro, half the memory of a DGX Spark, half the disk of DGX Spark, and no CUDA, is already more expensive than a spark. The Spark runs Linux natively, has a tiny form factor, and is usable as a general purpose computer.
I use a DGX Spark with NixOS on it as my daily driver and it's fantastic.
If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
> If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.
32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.
You are mistaken. I'm running Qwen 3.7 28B 4bit (MLX) with a 200k context window and everything total is 32GB RSS.
Is this the best? No. That's why I said the sweet spot. Getting from 16GB macs to 32GB is perhaps possible. Jumping to 64GB or 128GB as the default is simply unreasonable right now.
I have a similar machine, and briefly poked at running a local LLM, but got discouraged after a couple days. The quality, responsiveness, and impact on the rest of the system didn’t seem worth it to me.
What sorts of things are you doing with the local LLM? Anything interactive? Should I take another look?
Yes, 15-30 t/sec is pretty slow for local models so I recommend running local LLM tasks overnight where there isn't a risk of chewing through your token budget from a rogue loop or sub-agent. Even if it takes hours, you're sleeping anyway so no concern. herdr + pi works great for this but there are lots of harnesses.
I assume you mean Qwen 3.8-27B? Yes, you can run this in 32GB of RAM, but it's very context limited. With KV cache compression and other techniques, it's better now than in the past, but I'd still want more RAM, personally.
I run Qwen 3.8 27B just fine on my Mac mini M4 24GB. I use Unsloth's Q3 XXS with 128k context. It successfully completes long horizon tasks with OpenCode.
32GB of fast unified memory is enough for Qwen 3.8 27B.
- 16GB for the weights at Q4
- 9GB for the full 256K context at Q8
- 7GB spare for overhead and system.
The problem is that these Macs have 32GB of slow unified memory.
Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
You can, you just need a beefier PC, and it's more annoying in terms of noise and heat vs throwing something on your server closet. Plus you don't need to worry about other software stealing resources and whatnot.
Keep in mind that if you want MTP it adds a few gigs. If you use sub-agents it turns already slow generation into even slower generation. Won't be doing any compling (so rust, c and probably go are not avaiable) becase those add memory pressure during compiling.
32gb of unified memory is enough enough for system to be used for anything other than LLM generation.
> Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.
If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.
>massive PD disaggregated cluster of B300s connected via NVLink.
So a headless server.
Macs were mentioned because that's what the post is about. It could be a PC (I use a 2x3090 PC). The point is that it's a better experience to have a box dedicated to the LLM than running it in your system. Obviously in your home, so local.
Your agent(s) need to work on something, ie. running TypeScript, your app, your tests, Docker, Redis and/or database you need to run harness and user side apps ie. VScode, browser etc. it all adds up quickly.
Single user conversation spawns multiple parallel backend conversations, you need extra room for it as well, not just single context.
This plus usual apps like Mail, Spotify, iTerm2, SourceTree etc. also fill in memory.
Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.
I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
Huh, glad I grabbed my Neo two weeks ago. It's the "top" spec version, but still a good bit less than a MBA - seemed like a pretty reasonable replacement for the M1 iPadPro it replaced (wanted to go back to a normal laptop vs tablet).
I am missing something in the article. From what I am reading, AI companies are so desperate for memory they are buying Apple Systems and other hardware and striping them for RAM and maybe other components.
Apple hardware uses a unified architecture. That means the CPU and RAM are integrated together: it is very difficult/impossible to strip the RAM from these systems.
This unified architecture makes Apple hardware very good for AI work, where latency between RAM and CPU is very important
Even the SSDs (which are normally more strippable) use a proprietary hardware form factor.
I think the article mentioned the real draw:
> Apple noticeably promoted the ability to link multiple Mac Studios together into a single, more capable system for running large frontier AI models, a feature aimed at business and developer customers rather than everyday consumers.
(Couldn't read the source cited, might have more info: The Information)
Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
I need a new little Mac for my music studio, currently an M2 MacBook Pro. I thought I'd be fun to experiment with some local models as well. Well, let's price up an M5 Pro. $3,019 with 64GB RAM and a 1TB HD. Three thousand American dollars for a Mac Mini. Beefy spec for sure but not comically so.
Frankly even the entry price is a bit high - I remember buying one for my son a few years ago (M1 mini) and it was a few hundred; now we're up to $900 for the base model.
I bought a 5090 a year an a half ago for $2000. The same card, now a year and a half older, is $4000. Then there is the RAM - I bought 96GB, wishing it was 128, and now the price on my old RAM has doubled.
> Then there is the RAM - I bought 96GB, wishing it was 128, and now the price on my old RAM has doubled.
I also bought 96GB some while ago but after the initial increases, thinking I'll wait it out. Now 128GB is far more expensive than it was when I first looked. Luck has it I want DDR5 RDIMM as well, which seems the hardest hit when it comes to RAM prices, fun stuff.
Isn't OP complaining that the price went up because people want them for AI use? I guess it also went up because of cloud AI use increasing component costs, but those local models you want to run probably were trained in the cloud..
Isn’t a Mac mini annoying to use as an HTPC? You have to deal with a remote, software, and a full OS, compared with an Apple TV, which has a good remote and is optimized for TV use.
Isn’t a Mac mini annoying to use as an HTPC? You have to deal with a remote, software, and a full OS, compared with an Apple TV, which has a good remote and is optimized for TV use.
I've been using one for about a decade as a media server.
It just sits in the cabinet happily running the macOS TV program with the video files on an external hard drive. Playback on the TV is handled by the AppleTV's built-in Computer app. Works beautifully.
I have more movies and TV shows on that box than I could watch in my lifetime — a combination of ripped DVDs (Netflix, public library, and purchased) and OTA recordings.
When the cable goes out in my neighborhood (frequently), or a big storm screws up satellite reception (seasonally), I just don't care because I'm all localhost. As long as the lights stay on, everything is fine.
No ads. No privacy violation. No fees. No bandwidth congestion. No buffering. No subscription rate increases. All I pay for is electricity.
Now if only they hadn't discontinued the Mac Pro. Could be quite the AI machine with multiple compute GPUs at higher bandwidth than an external Thunderbolt enclosure.
This may sound a little wacky, but one potential use case I'm considering is robotics. Say you want to use a fine-tuned mid-size VLM model right on the robot. You pretty much have to use the Jetson line, which has a great ecosystem and everything (depth cameras, lidar, SLAM, small segmentation models, etc) runs on it, but it's gets very expensive very fast if you want to run LLMs on it. The Jetson Thor lines are 3-5k depending on memory etc.
One very efficient option today is to have the cheapest Jetson (Orin Nano) run the classical robotics stack, then have a base mac mini run nothing but the VLM. The Mac mini is considerably cheaper and faster at these workloads than the mid-range Jetsons.
I think this wonky situation is because Apple us under immense consumer pressure to absorb the ridiculous memory prices, while the Jetson is aimed at "business" and much more likely to fluctuate with the market. Last year I bought a Jetson Orin Nano 8GB for $375CAD, today that official nVidia Amazon page is out of stock and other sellers have it listed for $900-$1100CAD. Absolutely bonkers pricing.
There are so many used M4 Mac Minis for sale on marketplace for $600+ now that the people I guess got bored of the local models and decided to see if they could make their money back as they probably bought them when they were $399 on sale earlier this year. I lowball them every time.
I imagine Apple could also grow their business in the EU by marketing to companies who want powerful AI features but can't leverage 3rd party AI services due to GDPR. If only they could come up with iOS, App Store, and developer policies that respect the sovereignty of the EU.
The part where people starting buying Mac minis just to run OpenClaw was a pretty sudden movement in the market. These computers aren't powerful enough to run big local LLMs but are still being purchased for AI workflows.
A year ago you could get an M4 Mac mini for $399 on sale and now the same one used goes for over $700. The general AI RAM/SSD spike is part of that but there was also a huge demand spike for small, powerful, desktop machines that could be easily configured with these workflow tools.
Sad that Apple is stooping to the level of spamming the web with bots and fake news to advertise their products. Are there laws against this sort of thing?
So for people who don't understand, there are two markets for Apple hardware in this space:
1. Running an agent like OpenClaude. The $599 Mac Mini was an insanely good deal for this. I happened to buy a M5 Pro Mac Mini for $999 last year for other reasons. The equivalent is now almost $2000; and
2. Hardware for running inference on local models. This to me is the far more interesting market because Apple has a real opportunity to disrupt NVidia's stranglehold on the market.
With current architecture, the largest model you can reasonbly run is the amount of memory on the GPU and is a function of the quantization (eg int4, int8, fp8, fp16, etc) available and the number of parameters. NVidia aggressively segments the market. The most VRAM on a "consumer" card is 32GB on the 5090, which allows you to run ~31B parameter models.
In comparison, the RTX 6000 Pro has only slightly more CUDA units than a 5090 but has 80GB of VRAM. A few months ago they were $10-11k. Now they're ~$16k.
Macs use a shared memory architecture. Apple has previously sold Mac Studios with up to 512GB of RAM. Almost all of that memory can be used to hold much larger models without taking a penalty for interconnections between different GPUs or machines. Plus Apple interconnects between computers are actually relatively good by chaining TB5. It's still slow but it's about the best non-enterprise option available.
But the previous Mac Studios just didn't have the raw FLOPS and memory bandwidth. The M5 Ultras are up to 1.2TB/s of memory bandwidth. M3 Ultra had ~900GB/s. RTX 5090s and RTX 6000 Pros are 1.8TB/s. The current best HBM3 NVidia DC GPUs are at 3.2TB/s IIRC. But the M5 Ultra has a claimed ~4.5x the FLOPS of the M3 Ultra.
We don't have our hands on these yet but it probably means they are going to be much closer to a 5090. I expect ~50% of a 5090's inference speed. That may sound bad but it's actually really good because a 256/512GB Mac Studio can probably locally run the best Flash models. With NVidia hardware you'll need to spend many tens of thousands for that.
We'll see what the inference speed is but I expect it to be usable. DeepSeek v4 Flash, for example, will be entirely runnable. We're not at DeepSeek v4 Pro local yet.
> 1. Running an agent like OpenClaude. The $599 Mac Mini was an insanely good deal for this.
I still have zero clue how "Buy a $599 Mac Mini to have a sandboxed LLM API caller" became the default. If you're not doing local inference and don't need to inject into iMessage or iCloud, all you need to run openclaw-style harnesses that call external APIs is a Raspberry Pi 4B, an N100, an HTPC, or that 10 year old laptop sitting in your desk.
There is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded.
I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's zero locally.
Likewise. I have a huge demand personally to run AI noise-filtering models on many TB per month of raw video files. It takes about 3 days per file.
Apples ProRes codec is only licensed to run in high quality mode on a Mac, and so my Nvidia PC can’t do what I need. Thus, I own the beefiest Mac Studio you can currently buy. I would pay more for more TFlops.
I have done local LLM on there but it wasn’t interesting. Far worse performance and intelligence per dollar than the cloud boys.
There is no cloud offering for my video needs though.
It’s probably files that, over the course of a month, add up to multiple TBs.
Which would suggest that a 1Gbit fibre connection would be adequate. For serious commercial usage, multi Gbit fibre is available in many places around the world.
10mbps constant = ~3TiB a month, which is close to what I assume they're doing to acquire the video (that's about what a few SD x264 streams would run). So you don't even need a fast connection, per se.
Yes, basically like alphago. I’m teaching it to play magic: the gathering.
I had to start with some heuristic-based bots that played the decks very simply just to get to the point where the was some signal to learn from. I did behavioral cloning on the bots as a foundation, then self-play.
I’ve been happy training and running inference for small language models on my M4 Mac.
Inference with MLX is surprisingly zippy. I’m running a classification task on the entire HN comment dataset and it’s projected to take about two and a half days.
Yes, I could do it much more quickly by throwing Modal GPUs at it but, for low-priority work, why not use my underutilized M4 chip?
Same, but with vision models. Unfortunately, I might be at my limit locally. I have three models that I'm using to find and identify objects in pictures. The largest dataset and model now takes about 8 hours per epoch on my Mac M4 with 16G memory.
Yeah this was what got me to start doing short rentals of bigger gpus in the clouds, upload your parquet files and it takes a couple of hours for a thing that would have my mac at 100% for a couple of days
Is Modal at all similar to Vast.ai or just related because "It's for AI"? I looked at Modal's page for training, and it talks about using some SDK and other junk, can you not just get a beefy instance from Modal with tons of VRAM to do what you want with?
Do you find that CoreML manages to fill up your drive with so many tiny files that a reboot takes hours to clean them up? I keep meaning to get my friends still inside the spaceship to file a radar about that.
A C-level executive I know is getting a top-of-the-line new Mac simply to function as a personal build server and host for agentic coding instances - they are able to orchestrate so many parallel projects that they're hitting RAM limits from sessions and the builds and local test runs they're kicking off (largly unsupervised). Before AI, they'd only had a MacBook Air; this completely changes their workflows. They talk about how many other executives they've met are equally giddy at having gone from coding few to no projects themselves, to coding more projects in parallel than any of their respective pre-AI technical colleagues.
I'd suspect that agentic coding has birthed so many new effective engineers, that the entire dynamics of demand for high-end machines have been upended.
ram? coding instances? i suspect this is not doing any inference on the machine and i also suspect the ram use is due to a bagilion node/python processes running.
nothing like 600mb of for type check server another 500 for webpack and another 600 for the inevitable electron wrapper you didnt know about. python isnt as bad but still not great.
The general problem I have with this is that I think those C-level executive should be doing C-level stuff and not programmer stuff. If you're using AI to help with your main job, great! If you're using AI to do someone else's work, then I think maybe you should change jobs. If you want to be a programmer, go for it! But not at the expense of your "real" job.
I'm pretty sure the C-level people don't want us programmers faffing around with their roles. Why are they encroaching into ours? Don't they already have enough to do; setting the direction of the company, making sure it's profitable, selling things to people, etc.?
It very much depends what projects they are coding. One thing agentic coding is really good for is developing highly specific custom applications specifically for your own use case. This can include the kinds of decision support and analytics tasks a C-level relies on to understand how their organisation is functioning, what's actually going on, and where the gaps are.
My team is using coding agents to develop analytics and reporting applications, and an engineering automation application, specific to our needs, that simply would not exist otherwise. The cost and time would be far too great. I can easily imagine there are plenty of cases like this, for all sorts of roles, including managerial ones, where only the person doing the job understands the job well enough to specify a requirement and guide an agent to code exactly what they need.
Classic monopoly move: Control the user base, then control hardware. Any decent always-on local LLM setup with Apple devices will have to compete with these behemoths now. Great.
484 comments
[ 0.91 ms ] story [ 18.8 ms ] threadI can't even name one person who fits this mold, let alone a non-insignificant proportion of people. Who are you thinking of?
Your letter will help feed the boiler that heats the building and is much appreciated.
One thing to watch for when Apple introduces the new phones coming up shortly is whether or not Apple has replaced Qualcomm in their flagship smart phones because that is coming up soon Qualcomm has given warning to their investors.
Incidentally ios27 has a user control slider for glass frostiness.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
“Not as fully staffed as some people might hope” or “Developer Relations isn’t as responsive as I’d like” are both at least not obviously false.
[1]: https://www.youtube.com/watch?v=fYuH2Kl_b98 [2]: https://scholar.google.com/citations?view_op=view_citation&h...
Yes, the whole Deep Learning thing was luck, but as with most lucky things, they ensured they were positioned to capitalize on it.
AlexNet kicked off a new wave of research around neural networks by demonstrating they could be scaled well and trained on GPUs.
This is clearly a mis-statement, they have a whole annual conference for developers. Maybe they mean specifically AI devs.
I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
Bloody training data full of humans demanding sleep. I tells ya...
Is this Claude code? Or your local? I assume Claude? I'm more than a little staggered by this, like, it makes no sense! It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.) The LLM just... decided? I'd be so mad.
WTF indeed. Can one even file bugs?
Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.
Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.
But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)
I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).
What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.
When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.
I keep hoping that one day some comment is going to paste a link to some kind of idiot-proof guide or piece of software that’s “90% as good as Claude but running local.”
And by 90% I don’t mean that the model is 90% as good or runs 90% as fast, more like all the other stuff you mentioned is set up out of the box.
Though keep in mind not being beholden to shenanigans from said cloud companies (and interference from government entities!) is definitely worth something intangible.
I just ordered a new Mac Studio M5 Max 128GB $5899 ($6400 with tax) to be able to run a bigger "consumer size" model in the 70B parameter range (~96 GB). That said, I have no illusions that this expensive setup with a Qwen Flash coding LLM will be comparable to a $20/month subscription. Even upgrading to an even more expensive Mac Ultra 256GB for $10000 to hold a bigger model still won't be comparable. Apple hasn't shipped my Mac yet and I'm still considering cancelling it and downgrading to a smaller 64GB RAM config ($4299) to save $1600.
Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I use some local LLM for easy tasks or when I hit rate limits. No issues with privacy so that wasn't part of the motivation at all. I just wanted some local AI capability to augment a subscription. I've not totally convinced myself of the cost/benefit of this.
Based on today's consumer hardware landscape, you're paying very high prices for crippled capability compared to the cloud AI subscriptions. We're also in a transition period where the next iteration of hardware improvements have some compelling features for local AI. Apple's upcoming M7 (2027 or 2028) is anticipated to have better GPU and neural engine to help with prefill TTFT. AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc. Maybe apply my $1600 savings towards those newer products. Those future products are still very expensive but maybe the cost/benefit will be better.
If so, your opinion on anything technical can be safely ignored.
The maths don't check. With Deepseek Flash one goes a very long way with 1600$ - even 10$/month, for easy jobs, are more than 13 years, and at a higher quality.
The low hanging fruit stuff for me is more something I use it for because I have the local LLM setup running anyway. It wasn't the reason I bought it, but now that it's there I might just as well use it as much as I can.
No, this is a big misconception, and part of the cargo cult.
Use cases like the parent's are essentially about having a local LLM handle the leftover tasks. By that point, a lot of bits (main/big tasks) have already left home anyway.
- There's no guarantee of the $20/month service, and it likely has some limits compared to dedicated hardware token wise.
- Model are becoming more and more efficient, in many cases an M1 Max Mac Studio is still capable with 32 GB. 128 GB ram may not be the necessary baseline.
- Folks may think they want to only have a general model running locally (it's the comparable after all from the cloud providers), but we have to remember if the tasks we're trying to do ultimately are more specific than general and if there's space for the smaller models to do that.
Big upgrade to memory capacity but memory speed is only going up by a few percent, so its still going to be slow with more than a few B active params (I have one)
I think that some of the hardware design folks have been blindsided by AI demand and we haven’t really gotten that next generation AI hardware yet, to the point where buying M5 isn’t going to make sense in a couple of years.
Rumors seem to be that the M7 is the generation that Apple is looking to push AI performance much further.
I’m not sure that Apple anticipated this specific route that computer hardware has gone and I don’t think M5 and previous iterations were really specifically architected for local AI performance, more like they happened to be pretty good at it.
What specifically does that mean? The M5 series has 10 cores per 128 bits of memory bus; are the cores unable to keep up with the RAM? I thought they did and memory bandwidth was usually the bottleneck. But the memory bus is already very highly clocked and goes up to 1024 bits wide so it's hard to picture memory bandwidth having a huge leap.
[1] "LPDDR6 is coming." - https://news.ycombinator.com/item?id=49436849
50% more pins... we'll see. If they can reasonably make that fit then even more shame upon the traditional desktop CPU makers for sticking with 128 bits for so long.
I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.
What I noticed is that (1) the great local models are optimized run inference (diffusion & LLMs) well on 32GB VRAM <= GPU's because that that's what the target has ...
(2) The quality of local models (esp. in diffusion) is increasing faster than the need for more VRAM - additional reason for the value of these FAST GPUs to increase!
(3) RTX PRO 6000 96GB is really great for fine tunes (ai-toolkit) :) but doesn't outperform my RTX 5090 with inference by anything significant on the good local models.
I have never run an AI job on a Mac, i also have doubts about performance and compatibilities - since the reviews almost never compare directly.
What's an RTX 9070? Do you mean the RX 9070 or RTX 5070?
We also built some QA agents that are always playing our games from the same builds a player would and flagging things to fix/improve; that alone needs the game focused and front-and-center so it can properly screen-capture for deciding what inputs to take next (and for screenshots/replays), which also means we can't really do any hands-on work at all on the machine when it's running.
Having a separate (and tiny) machine for all of this has been great. We don't bother with local models because, you're right, the $20/month sub is way better than anything that can run on small consumer hardware atm.
That said I have an RTX 5090, not a Mac Mini, so it's not exactly the same level of performance... The latest open models run at 200 tpm at around 30B params.
What they are doing that you might not be is context caching, retaining a user profile about you in vector db, and adhering to strict output templates - often invoking multiple different prompts to assemble the end response to the user.
You’ll notice the mainstream ones always kinda answer in the same way. They are also able to mix code and images with text (multi-modality) because they are breaking down and routing your prompt to different models then assembling a result.
That’s all just software, has nothing to do with the model.
More people imo should be experimenting with it rather than chasing better models. This area is where all the differentiation is happening in the services, and what gives an AI provider like Gemini or Vidu or Singify an edge over running the equivalent locally
Yes there are smaller ones like mistral 7b vs larger ones which deliver only a slight improvement, but tbh most of the larger ones are worse than Mistral 7b.
The major providers like Google Gemini and ChatGPT and the “me too” company Anslopic are doing context caching (fact check me) and db retrieval to build responses.
Their responses adhere to templates and multiple different models are requested (one for the personalized description, one for the code solution, one for the image diagram, etc.) this is called multi-modality. Gemini and 4o stole most of their UX ideas from a project called Ragdoll (the first ever multimodal chat AI software released months before 4o).
You can make the same exact quality of AI chat service at home setting up harnessing like I described here namely the context caching and multi-modality. That is what differentiates an experience like Gemini from running local Ollama or Stable Diffusion or whatever where you are just getting a raw response from 1 model at a time.
I have an older M2 Mac mini that does the OCR and visual description of all my screenshots. Screenshots are stored on my NAS.
I like to screenshot things as a quick way to remember. They are things that I would not be comfortable sending a cloud provider (customer data, prototype screenshots, bank dispute details).
It runs Qwen3.5:9b and glm5.2-ocr with Ollama and uses about 10GB of RAM. It automatically releases the models from RAM after 5 minutes of inactivity so it is pretty seamless to leave running in the background.
All the details are stored in a simple webapp with a SQLite db that I can search through.
> They are things that I would not be comfortable sending a cloud provider
It's also an old machine that the commenter already has; it's intellectually dishonest to compare it to the price of a brand new, 4-iteration-newer machine.
Neither openai or xai. Anthropic maybe but not likely. Mistral is the most likely one because they're under the EU laws but I think they focus more on commercial these days.
a) model I pick will not 'suddenly' go away
b) I am sure my data stays where I want it
c) my inference mac can run other things if I need to
I pay for that.
Yes, of course the most economical path is to hand over all your data and become fully dependent on a cloud provider who is already operating as scale, hoping that they won't change/remove models, hamstring capabilities, or raise prices.
If this were a thread about hosting your own email or blog or cloud photos, you'd have plenty of people out here telling you how easy it is to do it yourself instead of relying on Gmail for email or WordPress/Medium/Substack for blogging, or iCloud for cloud photos.
And yet, without fail, every single thread about self hosting local models seems to have some copy/paste form of this cost-savings argument.
Where is the appreciation for this cool thing GP built? Where is the appreciation for the desire to figure out how to host your own version of the incredible capabilities that were not available merely a few years ago? And why, on this site of all places, would someone advocate trading all of the knowledge and independence gained from learning how to host something like this ourselves in favor of throwing it all over the wall to Google?
Come on.
It will be horrible to be dependent on an AI who is also be trying to sell you various goods and services.
We're going to need AI whose loyalty is to us and only us.
Because there are more privacy guarantees there, depending on the provider. "But what if they violate their contract!" is some pretty tin-foil hat stuff.
How is this any different than a business running their website out of the cloud, assuming you are using a provider with appropriate contractual terms?
You can care about tracking and ads but still be comfortable storing your backups in the cloud, and many have been for quite awhile, even sometimes without encryption - that is totally different than e.g. Meta actively trying to track you and understand your relationship graph and your purchases etc.
I’m pretty sure that all of those disclaimers that all the AI model makers have for you to sign off on to say that they’re not responsible for anything that might go wrong if your work gets copied accidentally and used someplace else wink wink?
You know the lawsuits for that particular aspect are incoming in the future…
But no way whatsoever I'm uploading my personal files, photos, emails, chats into cloud AI. No way.
In general, I'm a big believer in doing more with fewer resources, within reason, and think having local setups really helps me be mindful with what's happening under the hood with these systems and managing context efficiently to get high quality results.
A $20/month Gemini subscription is truly all you need, then yeah, sure.... obviously a homelab setup is a ridiculous alternative on a pure cost basis. For most people doing "real" work with LLMs 40+ hours per week, a more apt comparison would be one or multiple $200/month subscriptions. At which point the break-even point of a homelab is much sooner.
However, most people running homelabs are doing it for other reasons. Independence, learning, and/or privacy issues.
- Is that even worth the electricity price compared to api? - We don't ask that here
I have not spun up my own homelab, but, I have researched it and the electricity costs can be mitigated to a large extent.
One... the GPUs can be massively clocked down during idle state, to the point where the fans can be shut off as well. The machines themselves can be shut down and wait for a magic wake-on-LAN packet if needed.
Two... even under load, the GPU cores can be significantly underclocked with very little performance loss. The GPU and VRAM/HBM clocks are independent, and the GPU is largely bottlenecked on VRAM/HBM so you can just drop the GPU speed. A commonly reported figure I saw was, basically, 350W nominal cards being underclocked to consume "only" 200W under load with ~5-10% perf loss.
This all assumes you're using discrete GPUs and not AIO systems like a Mac Studio which is going to be pretty efficient just by design; they idle at 35W or so and in practice max out at a few hundred watts. I believe DGX Spark and Strix Halo are similar.
I must stress that this is second-hand anecdata here, admittedly, but my understanding is that it can be pretty manageable.
Doesn't Apple do this already within it's OS all locally? It certainly does it for OCR and categorization.
Works out really well.
You can grab the code here: https://github.com/mcotton/listing
I run it in Docker on my laptop or Synology NAS. It uses the OpenAI/Ollama URL schema for processing.
For OpenAI and Anthropic, the $100 subscriptions cost 5x the $20 subscriptions and give you 5x the tokens. And the $200 subscriptions are 10x the cost for 20x the tokens. (Tokens cost 50% as much.)
I've spent $2 in the last 2 weeks on OpenRouter. I've been trying to only use the medium sized models that I would otherwise be able to run on a nice local setup. That nice local setup would cost ~$4k. I don't know what the operating cost would be, but I would be concerned that my home electricity would cost more than at a datacenter. It just doesn't make sense right now except for privacy reasons.
I'm probably going to hoarde open weights models in the ~31B range until memory costs fall in a few years. Then, I'll buy some hardware to run at home just so I feel more sovereign over my stack regardless the cost/token speed.
But I am looking forward to lower hardware costs!
I realistically costs $5-10k to replicate a ChatGPT like agent. And it doesn't scale.
That's still really close. And models and quantization etc keep improving.
I'm absolutely positive that I'll be switching to mostly local AI in the next 5 years.
If you have a real product and can actually sell it, youre taking a largish risk relying on the cloud.
From model changes, alignment, to enshittification and the natural cognitive offloading, you could be one day removed and ROI tanked.
Think of AI like a mafia boss who helpfully supports you untill they need a favor. Thats all cloud AI is in America.
Fast 150+ tg and 2-8K pp Qwen 3.8 27B nvfp4 is about 8K (5090 +PC) Gives really only one concurrent session that flies because kv caching is not perfect for ninfer https://github.com/Neroued/ninfer
both are very serviceable, I prefer FP8 on 2xR9700
But, yes it doesn't scale that well but in 5 years the same hardware should still be very capable of running some great MoE models, for example Qwen 3.6 35BA3B on 5090 can fly at 600 tg
E.g., the M7 chip is rumored to be the one where Apple has poured really serious effort into local AI performance where previous generations seem to have mostly been coincidentally good at it.
Maybe this is an incorrect opinion but I don’t personally think that the M1-M3 or maybe M4 or even M5 chips were designed with LLM inference in mind at all. These were designed with things like video rendering, image/video ML, and rasterization performance in mind.
I recently built a minimal Dark Software Factory out of an N150 Mini PC. It uses three models; Sonnit, Sol, and Gemma.
But, I have a LOT of instructions about how I prefer the software it builds. Gemma doesn’t handle all my instructions very well. But it’s close!
I’m running gemma-4-12b because I have limited RAM and larger models were too slow.
I do two types of jobs: planning and prototyping. It has done fine at some of my planning rounds.
I still consider it experimental and don’t use it a lot but I think we’re getting there.
The model I’m specifically targeting to use at high speeds is Qwen 3.8 27b @q4ks. This model actually proved to be good at coding (it sits somewhere between Sonnet 5 and Opus 5 capability). M1 got 10 tok/s, Ryzen Halo 20tok/s, and Radeon 7900 XTX 50tok/s (can only do 128k context window in Radeon card).
The prefill gets extremely slow around 50k tokens in context window (whatever prompt processing stage entails could be wrong about phases here). It takes about 2 hours to fill the context.
Even with a drafter model intended for speed instead of mtp, I can’t get past 70tok/s, still is extremely slow to process prompts as context grows, and drops down to 40-50tok/s anyway making this config still moot for improvement on my Radeon card.
The only thing I can point to slowing me down is bandwidth of the card itself.
I am waiting to actually get my 5090 right now and I am betting that the 1700 Gbps of capacity will fix my prompt processing speeds. I don’t need full PCIe lane bandwidth to serve my house I just need to load the full model into vRAM and let the GPU do its thing.
Additional benefit to the external enclosure route is being able to migrate the inference between devices more easily. I can develop out the infrastructure then migrate the card to be hooked up to a shared node in the house with all the tools necessary for my family to take advantage of the privacy enhancement that comes with local inference.
Prior to two weeks ago, I was just using Pi and Ollama.
I have tried my hand at putting together a few harnesses and I finally landed on what I like. Been working on this small app to handle running llama-server for me from any device that has the llama-cpp stack setup: https://github.com/SamInTheShell/loom
Qwen 3.8 is the first model I've been using that hasn't been having issues doing edit calls. Here are my llama server settings and GUFF that I use: https://gist.github.com/SamInTheShell/0bf838e8dc5093583b688e...
Initial results boiled down as follows.
# lmstudio-community/qwen3.8-27b@q4_k_m decode falloff 104.3 tok/s @ 12,683 -> 55.8 tok/s @ 240,755 (53% retained) prefill falloff 3,274 tok/s -> 1,059 tok/s (32% retained)
# qwen3_8_27b_nvfp4.ninfer decode falloff 173.3 tok/s @ 11,867 -> 139.3 tok/s @ 225,710 (80% retained) prefill falloff 8,726 tok/s -> 2,816 tok/s (32% retained)
I should still have room for more performance on the table. I've not even touched the overclock settings on the GPU.
This is a really cool project, I'm going to have to get into what those 3 guys are doing... assuming it can be done with what I got.
I'm not surprised at all.
Context: I have a farm of DGX Sparks and several RTX 6000's, and can run very close to foundational models with ~2 sparks
So if I stay within 35B, especially MOE, my M5 Pro 64GB MBP can also run them well, and it can do plenty of other stuff too including gaming. While 256 GB with such RAM bandwidth and powerful GPU sounds like fun on paper, it doesn’t seem to be the next level compared to 64 GB
Really curious what people run on 256 GB Macs
if they are a heavy user, perhaps they string 4x together.
LLMs are harder: not much useful below 12B, and the 700B+ ones are really much better. Models like Qwen 3.8 27b show promise: in a few years pretty good local AI should be in reach for anyone willing to buy a $1000 computer (but who knows what your $20 sub buys you then).
My Mac can do ~200x realtime (1 hour takes 20s or so). I can do several thousand hours per day. Its pretty incredible
Not sure how much that qualifies as AI vs LLM usage, but it seems to work pretty good
Check out the fluid-ml library which packages this up for ANE very nicely.
Of course, 131k context at 4-bit quant is a trade off, but even then, it's VERY capable. It doesn't feel that far behind something like GPT 5.6 Luna.
Local coding is a step down but good enough for a lot of things if you have privacy concerns.
Wild oversimplification, and benchmarks vary widely, but I've read a lot of benchmarks suggesting that Qwen3.8-27B (xhigh effort) competes with near-frontier models at a lot of coding tasks. To the best of my understanding it's not going to run very feasibly in 16GB of VRAM at usable quants however.
r/LocalLLM and r/LocalLlama are noisy, but valuable sources of anecdata if you have the time (or the tokens, hah) to comb through them. You are going to see a lot of modest setups there, and also guys with $20K+ of hardware.
The two things (besides my bank account) that keep me from investing heavily in local are (1) we are not guaranteed to get a steady release of open models in the future (2) a lot of the "fun" stuff LLM stuff that interests me involves orchestrating lots of parallel agents, which of course multiples the hardware you need to achieve it.
For example, I've been having good results having both Sol and Opus review the same PR, and then I have them cross-review each others' PRs. A next step I'd like to consider is maybe having a swarm of Luna agents review the same PR and have them fight it out... maybe with Sol doing final arbitration? I suspect 5-10 Lunas might outperform a single Opus. Or maybe not. But at any rate, that would be impractical in a homelab without a pretty big hardware (or time) budget.
It's about not having to accept any of the stupid "terms" of the corporations. It's about doing things the big labs don't allow you to do, like cybersecurity stuff, or even just chatting with the AI about some wrongthink.
It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.
If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.
Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?
How much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
It's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.
They're letting everyone else pay for model training, with the obvious end of the road being open weights.
They're letting everyone else overspend on first-generation AI data centers before the chip industry has truly optimized for today's AI designs (which will reduce data center footprint).
They're letting everyone else play with and pioneer UI ideas.
Meanwhile all they've done is put a few toes in the water: "Apple Intelligence" which is barely anything, and adding AI acceleration to their GPUs and doing a bit of up-market marketing to AI.
So yeah they'll probably follow with a second generation Apple Intelligence that incorporates everything everyone else pioneered that worked, and an M7 or M8 line of chips with a GPU augmented with whatever approaches the other chip companies found worked best for running models... and built around the state of the art model architecture the industry finally converges on. By then RAM will probably be cheap again, so you'll get a 14" MacBook Pro with an M8 with a tensor-GPU and 512GiB of RAM.
How so? In tokens per second when running major open-weights models, or something else?
I use a DGX Spark with NixOS on it as my daily driver and it's fantastic.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.
32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.
Is this the best? No. That's why I said the sweet spot. Getting from 16GB macs to 32GB is perhaps possible. Jumping to 64GB or 128GB as the default is simply unreasonable right now.
What sorts of things are you doing with the local LLM? Anything interactive? Should I take another look?
And I don't remember to have been able to have pushed to 200k context Qwen 3.6. 3.8 is running on my RTX 5090.
https://news.ycombinator.com/item?id=49514141
64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.
- 16GB for the weights at Q4
- 9GB for the full 256K context at Q8
- 7GB spare for overhead and system.
The problem is that these Macs have 32GB of slow unified memory.
Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
32gb of unified memory is enough enough for system to be used for anything other than LLM generation.
What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.
If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.
So a headless server.
Macs were mentioned because that's what the post is about. It could be a PC (I use a 2x3090 PC). The point is that it's a better experience to have a box dedicated to the LLM than running it in your system. Obviously in your home, so local.
If someone wanted to use an 8x B300 as their daily driver...go ahead.
It would still be the best way to serve a given model.
Single user conversation spawns multiple parallel backend conversations, you need extra room for it as well, not just single context.
This plus usual apps like Mail, Spotify, iTerm2, SourceTree etc. also fill in memory.
https://www.canirun.ai (five months ago: https://news.ycombinator.com/item?id=47363754 377 comments)
I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
Is that what others read ?
This unified architecture makes Apple hardware very good for AI work, where latency between RAM and CPU is very important
Even the SSDs (which are normally more strippable) use a proprietary hardware form factor.
I think the article mentioned the real draw:
> Apple noticeably promoted the ability to link multiple Mac Studios together into a single, more capable system for running large frontier AI models, a feature aimed at business and developer customers rather than everyday consumers.
(Couldn't read the source cited, might have more info: The Information)
Frankly even the entry price is a bit high - I remember buying one for my son a few years ago (M1 mini) and it was a few hundred; now we're up to $900 for the base model.
The only reason for this huge speedbump is that chip makers have been dragging their feet for the last 10 years with "just enough" memory.
Stuff is crazy expensive.
I also bought 96GB some while ago but after the initial increases, thinking I'll wait it out. Now 128GB is far more expensive than it was when I first looked. Luck has it I want DDR5 RDIMM as well, which seems the hardest hit when it comes to RAM prices, fun stuff.
I've been using one for about a decade as a media server.
It just sits in the cabinet happily running the macOS TV program with the video files on an external hard drive. Playback on the TV is handled by the AppleTV's built-in Computer app. Works beautifully.
I have more movies and TV shows on that box than I could watch in my lifetime — a combination of ripped DVDs (Netflix, public library, and purchased) and OTA recordings.
When the cable goes out in my neighborhood (frequently), or a big storm screws up satellite reception (seasonally), I just don't care because I'm all localhost. As long as the lights stay on, everything is fine.
No ads. No privacy violation. No fees. No bandwidth congestion. No buffering. No subscription rate increases. All I pay for is electricity.
What planet do you live on?
One very efficient option today is to have the cheapest Jetson (Orin Nano) run the classical robotics stack, then have a base mac mini run nothing but the VLM. The Mac mini is considerably cheaper and faster at these workloads than the mid-range Jetsons.
I think this wonky situation is because Apple us under immense consumer pressure to absorb the ridiculous memory prices, while the Jetson is aimed at "business" and much more likely to fluctuate with the market. Last year I bought a Jetson Orin Nano 8GB for $375CAD, today that official nVidia Amazon page is out of stock and other sellers have it listed for $900-$1100CAD. Absolutely bonkers pricing.
And?
A year ago you could get an M4 Mac mini for $399 on sale and now the same one used goes for over $700. The general AI RAM/SSD spike is part of that but there was also a huge demand spike for small, powerful, desktop machines that could be easily configured with these workflow tools.
1. Running an agent like OpenClaude. The $599 Mac Mini was an insanely good deal for this. I happened to buy a M5 Pro Mac Mini for $999 last year for other reasons. The equivalent is now almost $2000; and
2. Hardware for running inference on local models. This to me is the far more interesting market because Apple has a real opportunity to disrupt NVidia's stranglehold on the market.
With current architecture, the largest model you can reasonbly run is the amount of memory on the GPU and is a function of the quantization (eg int4, int8, fp8, fp16, etc) available and the number of parameters. NVidia aggressively segments the market. The most VRAM on a "consumer" card is 32GB on the 5090, which allows you to run ~31B parameter models.
In comparison, the RTX 6000 Pro has only slightly more CUDA units than a 5090 but has 80GB of VRAM. A few months ago they were $10-11k. Now they're ~$16k.
Macs use a shared memory architecture. Apple has previously sold Mac Studios with up to 512GB of RAM. Almost all of that memory can be used to hold much larger models without taking a penalty for interconnections between different GPUs or machines. Plus Apple interconnects between computers are actually relatively good by chaining TB5. It's still slow but it's about the best non-enterprise option available.
But the previous Mac Studios just didn't have the raw FLOPS and memory bandwidth. The M5 Ultras are up to 1.2TB/s of memory bandwidth. M3 Ultra had ~900GB/s. RTX 5090s and RTX 6000 Pros are 1.8TB/s. The current best HBM3 NVidia DC GPUs are at 3.2TB/s IIRC. But the M5 Ultra has a claimed ~4.5x the FLOPS of the M3 Ultra.
We don't have our hands on these yet but it probably means they are going to be much closer to a 5090. I expect ~50% of a 5090's inference speed. That may sound bad but it's actually really good because a 256/512GB Mac Studio can probably locally run the best Flash models. With NVidia hardware you'll need to spend many tens of thousands for that.
We'll see what the inference speed is but I expect it to be usable. DeepSeek v4 Flash, for example, will be entirely runnable. We're not at DeepSeek v4 Pro local yet.
I still have zero clue how "Buy a $599 Mac Mini to have a sandboxed LLM API caller" became the default. If you're not doing local inference and don't need to inject into iMessage or iCloud, all you need to run openclaw-style harnesses that call external APIs is a Raspberry Pi 4B, an N100, an HTPC, or that 10 year old laptop sitting in your desk.
I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's zero locally.
Apples ProRes codec is only licensed to run in high quality mode on a Mac, and so my Nvidia PC can’t do what I need. Thus, I own the beefiest Mac Studio you can currently buy. I would pay more for more TFlops.
I have done local LLM on there but it wasn’t interesting. Far worse performance and intelligence per dollar than the cloud boys.
There is no cloud offering for my video needs though.
It’s probably files that, over the course of a month, add up to multiple TBs.
Which would suggest that a 1Gbit fibre connection would be adequate. For serious commercial usage, multi Gbit fibre is available in many places around the world.
I had to start with some heuristic-based bots that played the decks very simply just to get to the point where the was some signal to learn from. I did behavioral cloning on the bots as a foundation, then self-play.
If you can fit it on a GPU, and especially for training, it is so much quicker than a Mac.
Inference with MLX is surprisingly zippy. I’m running a classification task on the entire HN comment dataset and it’s projected to take about two and a half days.
Yes, I could do it much more quickly by throwing Modal GPUs at it but, for low-priority work, why not use my underutilized M4 chip?
Modal significantly improves this. Highly recommend.
Do you find that CoreML manages to fill up your drive with so many tiny files that a reboot takes hours to clean them up? I keep meaning to get my friends still inside the spaceship to file a radar about that.
What game are you building?
I'd suspect that agentic coding has birthed so many new effective engineers, that the entire dynamics of demand for high-end machines have been upended.
nothing like 600mb of for type check server another 500 for webpack and another 600 for the inevitable electron wrapper you didnt know about. python isnt as bad but still not great.
I'm pretty sure the C-level people don't want us programmers faffing around with their roles. Why are they encroaching into ours? Don't they already have enough to do; setting the direction of the company, making sure it's profitable, selling things to people, etc.?
My team is using coding agents to develop analytics and reporting applications, and an engineering automation application, specific to our needs, that simply would not exist otherwise. The cost and time would be far too great. I can easily imagine there are plenty of cases like this, for all sorts of roles, including managerial ones, where only the person doing the job understands the job well enough to specify a requirement and guide an agent to code exactly what they need.
Local inference solves so many of the privacy and inconsistency problems with these frontier subscriptions.