Show HN: The fastest way to run Mixtral 8x7B on Apple Silicon Macs

18 points by woadwarrior01 ↗ HN
I’d originally launched my app: Private LLM[1][2] on HN around 10 months ago, with a single RedPajama Chat 3B model. The app has come a long way since then. About a month ago, I added support for 4-bit OmniQuant quantized Mixtral 8x7B Instruct model, and it seems to outperform Q4 models at inference speed and Q8 models at text generation quality, while consuming only about 24GB of RAM[3] at 8k context length. The trick is: a) to use a better quantization algorithm and b) to use unquantized embeddings and the MoE gates (the overhead is quite small).

Other notable features include many more downloadable models, support for App Intents (Siri, Apple Shortcuts), on-device grammar correction, summarization etc with macOS services and an iOS version (universal app), also with many smaller downloadable models and support for App Intents. There's a small community of users building and sharing LLM based shortcuts on the App's discord.

Last week, I also shipped support for the bilingual Yi-34B Chat model, which consumes ~18GB of RAM. iOS users and users with low memory Macs can download the related Yi-6B Chat model.

Unlike most popular offline LLM apps out there, this app uses mlc-llm for inference and not llama.cpp. Also, all models in the app are quantized with OmniQuant[4] quantization and not RTN quantization.

[1]: https://privatellm.app/

[2]: https://apps.apple.com/us/app/private-llm-local-ai-chatbot/id6448106860

[3]: https://www.youtube.com/watch?v=4AE8yXIWSAA

[4]: https://arxiv.org/abs/2308.13137

22 comments

[ 4.9 ms ] story [ 58.8 ms ] thread
Thanks for making this app! I've been using it for the past week or so. I've played around with some open source stuff but for some reason it's been enough hassle that I haven't stuck with it.

The "this conversation is completely private" messages are also oddly reassuring.

Thanks for trying it out! Also, please feel free to message me if you've got any feature or model requests.
I've been using Private LLM a few weeks now, it's certainly the best of the local options on an iPhone.

I'm wondering if you're going to add support for downloading custom models from HF? (e.g. mlx-community models).

Thanks! One significant reason why the app works as well as it does is OmniQuant quantization. It takes a few hours (depending on the size of the model) to quantize the model with OmniQuant, using an NVIDIA GPU. Using RTN quantized models from mlx-community or perhaps TheBloke on HF, would nullify that advantage.

OTOH, OmniQuant is open source and if users can quantize their own models with it, they should be able to side load it. I think this is something that we can add in the near future.

That would be really neat. I’d happily leave my server (RTX3090) or MacBook chugging away for a few hours to get the models I’d like on my phone. Thanks for your reply!
Yeah! That's pretty much my quantization workflow for small models. I've got a dual 3090 box under my desk to quantize 7B and smaller models and rent machines on vast.ai for larger models which need > 24GB of RAM for larger models (Mixtral needs an 80GB A100, for instance).
Just a nitpick - the website for the app flashes the UI for a split second and then goes to white screen (except the banner at the top to install the app from the app store) on iOS Safari.
Thanks for reporting that! Could you tell me what device this was on? I've tried it on a couple of different iOS devices (my iPhone 15 Pro, an iPhone 15 and an iPhone 13 mini) with iOS Safari and Firefox Focus, but couldn't reproduce it. As you can probably tell, I'm not a front-end developer. :)
Updating this comment, because I just figured out the root cause, and it is as silly as it can get. Leaving the original comment below the horizontal line just in case.

TLDR: your website wasn’t at fault itself (well, i think the error handling could’ve been much better with an explicit failure, but that’s just nitpicking).

The issue was that the link to the website (the one in the OP) would lead me to `https://privatellm.app/%3C/p%3E` (with the extraneous part decoding to a closing tag </p>) instead of just `https://privatellm.app/`. Once I went down the rabbithole, I noticed the URL, manually changed it, and the website worked.

Decided to repro it by going back to the HN post, clicking the link, and yep, the website failed again. Looks like there was zero fault of your own, and it is a bug in the HN reader app I use does this for that specific URL.

———————————————

It was on iPhone 15 Max Pro, iOS 17.3.1. I rebooted the device just for posterity today, still reproducing the same behavior.

Seems like it might be JS-related, because I finally made out what the split-second flashing UI says. It is a giant “Oh no, you stumbled upon a rarity” message, with a purple “go back home” button.

After it flashes and i see the white screen, i check HTML and it is pretty much empty (aside from a bunch of script tags in the body), which is why I suspect it is JS related and not CSS.

The dev console is spewing errors, but it isnt super helpful due to the JS bundle being minified[0].

0. I use Web Inspector extension for mobile Safari to check on network calls/js console/inspect DOM. If someone is looking for something like chrome dev tools on mobile Safari, that’s what I use and can recommend. Maybe there is something better out there now, but that’s the one I discovered a few months ago and found the be a solid “all in one package” (e.g., there is an extension I’ve used prior that does a slightly better job of inspecting DOM, but it cannot do anything else like JS console/network requests/etcH

Thanks for such a detailed report! The link to the website in the OP doesn't seem to be clickable for me on both Safari and Mobile Safari. I think I'll just explicitly add a redirect from `/%3C/p%3E` to `/` immediately and then fix the broken 404 page next. Thanks again!
No worries at all, and thanks for the app, i like it quite a bit.

I wouldnt recommend adding that redirect explicitly, and instead just make all invalid url paths redirect to the home page. Mostly because I am certain that the issue with the link for me is happening due to the HN reader app I am using and how it renders URLs (which is how they are clickable for me in the first place).

My mac book air is m1, 8G. Can it run this app?
Maybe at a small quantization, but you probably don’t have enough RAM.
Yeah, some of the bigger models won't work, but there are a bunch that will.
The amount of RAM is the bottleneck for LLM inference on Apple Silicon. With 8GB of RAM, 3B models StableLM 3B and smaller work well. 7B models nominally work, but there won't be much room left for running anything else. 16GB Macs are somewhat of a sweet spot, you can run 10.7B and 13B models, which are quite good.
I just bought the Mac version.

Initial feedback:

It would be nice to have a settings icon before the delete icon. I had to go to the bar menu to find that.

- In the list of models, it would be nice to detect the system RAM and highlight the ones that cannot run, showing how much RAM they would need.

- It would be great to run the same prompt with multiple models. If this would take time and loop through, the use case would evaluate different models using the same prompt.

I like the UI and the speed, good luck!

Thanks for buying the app and the feedback!

> It would be nice to have a settings icon before the delete icon. I had to go to the bar menu to find that.

This is great feedback! It's already the case with the iOS app, I'll port it over to the Mac app in the next update.

> - In the list of models, it would be nice to detect the system RAM and highlight the ones that cannot run, showing how much RAM they would need.

The app does a bit of this already. It does know allow users to bigger models (> 7B) on 8GB Macs and Intel macs for instance. Also, when that's the case it shows an info icon that informs them why they can't download the model. There's always room to improve the UX for this.

> - It would be great to run the same prompt with multiple models. If this would take time and loop through, the use case would evaluate different models using the same prompt.

This might be a bit more difficult to implement. Let me think over it for a while and see what I can do.

PS: You don't need to buy the iOS app separately, it's a free download if you've bought the Mac app (and vice-versa), because it's a universal app.

> Private LLM Works Anywhere, Anytime!

> The item you've requested is not currently available in the UK store.

Edit: that’s from the link in your post. The link from your site seems to work.

Interesting, and thanks for reporting! Are you by any chance, using the "HN reader app" like someone else was on this thread[1]?

[1]: https://news.ycombinator.com/item?id=39971102

I’m using “Hack”, and interestingly I can’t follow the link in your response either.

Edit: the link works, but just takes me to your OP.

I know the app is private but the “submit” icon looks like the “location services” icon that pops up in iOS when you are being tracked and the button on the keyboard says “send” as if it is being sent somewhere else. Just an observation!
That's a great observation! I wonder if a Apple Messages style "up arrow" (arrow.up.circle.fill in SFSymbol) within the TextField would be a better UI metaphor?