HN Explorer

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac (github.com)

919 points by gitpusher42 ↗ HN
Hi HN,

I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.

I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory.

The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are included.

The trick is to keep the shared part of the model and the KV cache in RAM, then stream only the routed experts needed for each token from SSD. An SSD is way slower than RAM, so the runtime uses a small expert cache and bounded parallel `pread`. While those reads are in flight, the GPU runs the shared part of the layer.

I ran more than 100 experiments. Most didn’t work. A few got me here. The experiments are described in the GitHub repo.

It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.

I also added an experimental OpenAI-compatible local server. It supports streaming and tool calls, and reuses one prompt prefix from the KV cache.

Try it! The Mac app is easy to install. On the first run, it will download 15 GB of weights from Hugging Face. The model is surprisingly capable.

I would love any kind of feedback!

133 comments

[ 0.31 ms ] story [ 39.4 ms ] thread
yakupov_bulat
Wow, amazing!

What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.

addaon
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.

Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...

gitpusher42
I am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb.

It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing

mxmlnkn
This sounds really cool. My intuition was that the selected experts might change heavily for each token, resulting in slow SSD loads for each token. This seems to be wrong. Did you create some statistics on how often the experts need to be changed? What is the longest token run without any expert change? What does such a token run look like? In which cases do experts change frequently?
znpy
I wonder if i can run this on my MacBook Neo!
h2aichat
Hope you can do it for Windows users also (and small graphics cards). Thanks
touwer
Cool! Is there any info on this doing harm to the SSD? (Or other parts?)
sscarduzio
How does this compare to DwarfStar4?
gitpusher42
uh, I don't think it is possible to compare them. DwarfStar4 is for high end macs and a lot of ram. this project is more targeted to low end devices and "general use" Gemma4 model
hsienchuc
I've run local video generation models on an 8GB graphics card and know firsthand that nothing runs smoothly when memory is insufficient. So seeing 14GB of weights crammed into 2GB of RAM is impressive.

If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?

Great work, congratulations on the release!

gitpusher42
Thank you very much!

I think it will throttle quite soon, but I haven't tried runs longer than 30minutes with this engine.

However, there is no constant load on ssd or gpu. i/o and gpu work are alternating and there is a brief idle periods for each i/o and gpu during inference (because gpu waits for i/o and after that i/o waits for gpu)

tredre3
I'm curious how your project compares to plain mmap!

Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).

It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.

greggh
It does exactly what it says it does. On my Mac mini M4 with 16GB of ram it is running at just over 5 tok/s. That jump from M4 to M5 is crazy.
mmastrac
I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other.

Feel free to reach out.

(currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)

nl
I tried out one of the NVidia diffusion models, and from memory it only worked on MLX but seemed to leave a lot of features out. Would your work support non-Gemma models too?
maxignol
I'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released. Great work !

Anyone got recommendation about what local model to use for what purpose ? I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.

[1] : https://github.com/JustVugg/colibri [2] : https://fireworks.ai/blog/kimik3-fable

huangsemao
What part of the optimization process gave you the biggest speed gain?
gitpusher42
Switching from mmap to parallel pread. From 0.5tok/sec to almost 4tok/sec. Running GPU work while reading missed experts also helped a lot, 4.4 -> 4.7
tracker1
This is actually very similar to some ideas I've been having for a while... that having a smaller entry model that knows enough about "expert" models that themselves are smaller to hand work over to could be better/faster/lighter in terms of working through real problems vs the megalith ones we currently use. Highly distilled experts and coordination with a fallback mode to a larger model option.
gitpusher42
afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use only these experts for generation. It doesn't require fitting full model into memory or per token ssd streaming
[deleted]
(comment deleted)
owaislone
Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.
rcarmo
Would be awesome if it ran Qwen (the MoE probably won't squeeze that low, but...). This because I have hardly been able to use Gemma for any sort of useful coding.
xenonite
With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with

  opts.languageVersion = .version4_0
or surround them with

  if #available(macOS 26.0, *) {
    opts.languageVersion = .version4_0
  }
You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)
quasarj
Why are you still on 15?
dudefeliciano
me too, everyone says it sucks and to wait for Golden Gate to release.
tedd4u
To build this (on macOS 15) I also had to do this (in Package.swift):

    @@ -4,8 +4,7 @@ import PackageDescription
     let package = Package(
         name: "TurboFieldfare",
         platforms: [
    -        .macOS(.v26),
    -        .iOS(.v26),
    +        .macOS(.v15)
         ],
         products: [
             .library(name: "TurboFieldfare", targets: ["TurboFieldfare"]),
cyanregiment
You’re a mad man - thank you!

Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?

gitpusher42
Thank you!

afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case

docheinestages
Is there a pipeline or approach to do this to any model? I'm particularly interested in Qwen 3.6 27B as it's the best for its size at the moment.
gitpusher42
This approach will only work for MoE models. There is a Qwen 35b-a3b. You just need to do GPU stop after router and read the requested experts to ram. And it is possible to build similar engine for this model (or feel free to adopt my engine) Not sure about generic approach for now, but coding with ai agents is relatively cheap now, you can try it
tpurves
Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two?

my impression right now is that M5 gen is on the cusp of practicality for local inference.

If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainstream.

Joeri
Without fundamental model architecture improvements the practicality largely depends on how Apple increases memory bandwidth.

Memory bandwidths (* = rumored):

  M1:       68 GB/s
  M2:       100 GB/s
  M2 pro:   200 GB/s
  M2 max:   400 GB/s
  M2 ultra: 800 GB/s
  M5:       153 GB/s
  M5 pro:   307 GB/s
  M5 max:   460 GB/s
  M6:       200 GB/s*
  M7:       240 GB/s*
  
  Nvidia 4090 1008 GB/s
  Nvidia H100 3.35 TB/s
Basically what we're looking at by the M7 generation is a tier shift, where the base M7 can do what the M2 pro did, and every tier moves up accordingly, with the M7 ultra becoming competitive with nvidia dedicated consumer hardware.
hacket04
I think it's M5 MacBook Pro, not M5 Pro, as he mentioned.
weras
Rushing to try it!
minraws
I have been working on doing the same for ling-3.0 seems very usable on my 5070 Ti now since it's only 5.1B active, you can even get pretty greedy and keep around 6% of each expert in memory and load the prompt and make the changes.
gitpusher42
uh, it's a bit difficult to discuss the classical approach with vRAM and regular RAM. Not really familiar with optimisations and hacks, I always worked with apple platforms and shared memory. But description sounds cool, good luck with your project!