Show HN: The fastest way to run Mixtral 8x7B on Apple Silicon Macs
Other notable features include many more downloadable models, support for App Intents (Siri, Apple Shortcuts), on-device grammar correction, summarization etc with macOS services and an iOS version (universal app), also with many smaller downloadable models and support for App Intents. There's a small community of users building and sharing LLM based shortcuts on the App's discord.
Last week, I also shipped support for the bilingual Yi-34B Chat model, which consumes ~18GB of RAM. iOS users and users with low memory Macs can download the related Yi-6B Chat model.
Unlike most popular offline LLM apps out there, this app uses mlc-llm for inference and not llama.cpp. Also, all models in the app are quantized with OmniQuant[4] quantization and not RTN quantization.
[1]: https://privatellm.app/
[2]: https://apps.apple.com/us/app/private-llm-local-ai-chatbot/id6448106860
[3]: https://www.youtube.com/watch?v=4AE8yXIWSAA
[4]: https://arxiv.org/abs/2308.13137
22 comments
[ 4.9 ms ] story [ 58.8 ms ] threadThe "this conversation is completely private" messages are also oddly reassuring.
I'm wondering if you're going to add support for downloading custom models from HF? (e.g. mlx-community models).
OTOH, OmniQuant is open source and if users can quantize their own models with it, they should be able to side load it. I think this is something that we can add in the near future.
TLDR: your website wasn’t at fault itself (well, i think the error handling could’ve been much better with an explicit failure, but that’s just nitpicking).
The issue was that the link to the website (the one in the OP) would lead me to `https://privatellm.app/%3C/p%3E` (with the extraneous part decoding to a closing tag </p>) instead of just `https://privatellm.app/`. Once I went down the rabbithole, I noticed the URL, manually changed it, and the website worked.
Decided to repro it by going back to the HN post, clicking the link, and yep, the website failed again. Looks like there was zero fault of your own, and it is a bug in the HN reader app I use does this for that specific URL.
———————————————
It was on iPhone 15 Max Pro, iOS 17.3.1. I rebooted the device just for posterity today, still reproducing the same behavior.
Seems like it might be JS-related, because I finally made out what the split-second flashing UI says. It is a giant “Oh no, you stumbled upon a rarity” message, with a purple “go back home” button.
After it flashes and i see the white screen, i check HTML and it is pretty much empty (aside from a bunch of script tags in the body), which is why I suspect it is JS related and not CSS.
The dev console is spewing errors, but it isnt super helpful due to the JS bundle being minified[0].
0. I use Web Inspector extension for mobile Safari to check on network calls/js console/inspect DOM. If someone is looking for something like chrome dev tools on mobile Safari, that’s what I use and can recommend. Maybe there is something better out there now, but that’s the one I discovered a few months ago and found the be a solid “all in one package” (e.g., there is an extension I’ve used prior that does a slightly better job of inspecting DOM, but it cannot do anything else like JS console/network requests/etcH
I wouldnt recommend adding that redirect explicitly, and instead just make all invalid url paths redirect to the home page. Mostly because I am certain that the issue with the link for me is happening due to the HN reader app I am using and how it renders URLs (which is how they are clickable for me in the first place).
Initial feedback:
It would be nice to have a settings icon before the delete icon. I had to go to the bar menu to find that.
- In the list of models, it would be nice to detect the system RAM and highlight the ones that cannot run, showing how much RAM they would need.
- It would be great to run the same prompt with multiple models. If this would take time and loop through, the use case would evaluate different models using the same prompt.
I like the UI and the speed, good luck!
> It would be nice to have a settings icon before the delete icon. I had to go to the bar menu to find that.
This is great feedback! It's already the case with the iOS app, I'll port it over to the Mac app in the next update.
> - In the list of models, it would be nice to detect the system RAM and highlight the ones that cannot run, showing how much RAM they would need.
The app does a bit of this already. It does know allow users to bigger models (> 7B) on 8GB Macs and Intel macs for instance. Also, when that's the case it shows an info icon that informs them why they can't download the model. There's always room to improve the UX for this.
> - It would be great to run the same prompt with multiple models. If this would take time and loop through, the use case would evaluate different models using the same prompt.
This might be a bit more difficult to implement. Let me think over it for a while and see what I can do.
PS: You don't need to buy the iOS app separately, it's a free download if you've bought the Mac app (and vice-versa), because it's a universal app.
> The item you've requested is not currently available in the UK store.
Edit: that’s from the link in your post. The link from your site seems to work.
[1]: https://news.ycombinator.com/item?id=39971102
Edit: the link works, but just takes me to your OP.