HN Explorer

Show HN: Gemma 4 Multimodal Fine-Tuner for Apple Silicon (github.com)

235 points by MediaSquirrel ↗ HN
About six months ago, I started working on a project to fine-tune Whisper locally on my M2 Ultra Mac Studio with a limited compute budget. I got into it. The problem I had at the time was I had 15,000 hours of audio data in Google Cloud Storage, and there was no way I could fit all the audio onto my local machine, so I built a system to stream data from my GCS to my machine during training.

Gemma 3n came out, so I added that. Kinda went nuts, tbh.

Then I put it on the shelf.

When Gemma 4 came out a few days ago, I dusted it off, cleaned it up, broke out the Gemma part from the Whisper fine-tuning and added support for Gemma 4.

I'm presenting it for you here today to play with, fork and improve upon.

One thing I have learned so far: It's very easy to OOM when you fine-tune on longer sequences! My local Mac Studio has 64GB RAM, so I run out of memory constantly.

Anywho, given how much interest there is in Gemma 4, and frankly, the fact that you can't really do audio fine-tuning with MLX, that's really the reason this exists (in addition to my personal interest). I would have preferred to use MLX and not have had to make this, but here we are. Welcome to my little side quest.

And so I made this. I hope you have as much fun using it as I had fun making it.

-Matt

15 comments

[ 179 ms ] story [ 313 ms ] thread
dsabanin
Thanks for doing this. Looks interesting, I'm going to check it out soon.
craze3
Nice! I've been wanting to try local audio fine-tuning. Hopefully it works with music vocals too
LuxBennu
I run whisper large-v3 on an m2 max 96gb and even with just inference the memory gets tight on longer audio, can only imagine what fine-tuning looks like. Does the 64gb vs 96gb make a meaningful difference for gemma 4 fine-tuning or does it just push the oom wall back a bit? Been wanting to try local fine-tuning on apple silicon but the tooling gap has kept me on inference only so far.
yousifa
This is super cool, will definitely try it out! Nice work
conception
I’m pretty excited about the edge gallery ios app with gemma 4 on it but it seems like they hobbled it, not giving access to intents and you have to write custom plugins for web search, etc. Does anyone have a favorite way to run these usefully? ChatMCP works pretty well but only supports models via api.
neonstatic
Just a heads up, that I found NVIDIA Parakeet to be way better than Whisper - faster, uses less compute, the output is better, and there are more options for the output. I am using parakeet-mlx from the command line. Check it out!
mandeepj
> I had 15,000 hours of audio data

do you really need that much data for fine-tuning?

sails
> Accent, dialect, and low-resource language adaptation — adapt a base Gemma model to underrepresented voices and languages with your own labeled audio.

Is this for TTS? Have been looking for something to do a local fine tune to get a specific accent

m3kw9
will it work for 32gb?
SuperRat-Beta
One thing worth mentioning for anyone loading model files from external sources: GGUF and pickle-based formats (.bin, older .pt files) have been attack vectors — CVE-2024-34359 was RCE via crafted GGUF in llama_cpp_python (CVSS 9.8).

safetensors was created specifically to address this. Worth sticking to it and verifying checksums when loading community checkpoints. torch.load(..., weights_only=True) also helps for .pt files (PyTorch ≥ 1.13).

Just added a full section on this to a reference I maintain on AI toolchain attack techniques: https://github.com/XiaoYiWeio/ai-agent-attack-techniques