Show HN: macOS GUI for running LLMs locally (avapls.com)
I've been working on this project for a while, and it has been in an "open" beta for some time. I finally believe it's ready for its first release.
I hope you like it.
Here are some potential questions that may arise:
1. How does it compare to LM Studio? It's likely that if you're already using LM Studio, you'll continue to do so. This project is designed to be more user-friendly.
2. Is it open-source? No, it is not.
3. Does it use any open-source libraries? Yes, it uses llama.cpp and a few others, as indicated in the license information included with the application.
4. Why is not using electronjs? Two reasons, I wanted total control over the whole tech-stack and second, I wanted to be able to send this to my friends over iMessage.
5. Does it support Intel macs? It should, but I couldn't test it.
6. Does it support older macOS? 12.6 is the lowest version at the moment.
7. Is XXX a bug? Probably :)
50 comments
[ 3.4 ms ] story [ 100 ms ] threadI would love to hear how you are using Ollama. One of the upcoming releases will involve an official release of Ollama on Linux with CUDA support of the box. From there, we will publish Ollama Docker images to enable GPU support as well.
See https://news.ycombinator.com/item?id=37562287 - I can't tell much more yet, but I feel like LLMs are way more useful than for a chat-bot or a vscode extension.
Only a few download links are baked-in at the moment but whatever *.gguf file you put in your Downloads folder should appear in the dropdown.
> A language server is a specialized program that processes language-related tasks. This includes activities like text generation, grammar correction, rephrasing, summarization, data extraction, and more.
This confused me initially. I've never heard language server in this context. To me, it has always been in the context of the LSP[0]. But maybe this is a common usage of the term and I've just missed it?
Even if so, it seems like an odd term, since this is a self-contained desktop application and not really a server, right?
0: https://en.wikipedia.org/wiki/Language_Server_Protocol
On the bright side, now that you know what a language server is, you can rename your thing so it can be searched for.
Why not?
As to Why --> this one actually has a decent GUI and one that's not (visibly) based on a locally running webserver -- so it seems like it's got a shot at getting 100x popular with just a YouTube video (or gasp Tiktok). With the popularity of open-source today I find it a little hard to trust that something is "ensuring maximum privacy" as you say without having the open-source badge (yes, I know I shouldn't be so naive, but no, I am not going to look at network level logs to find out if the app is truly contained to local).
IMHO, at some point, it's just about trust.
Now, to be 100% honest, I'd love to release this as OSS, but my current business model is based on freemium. And, sell it as white-label for companies.
EDIT: if you have better idea, I'd love to hear about it.
I'm not saying change your ways, just consider being a bit less flippant when you say things like "IMHO, at some point, it's just about trust". As someone who seeks to profit from this, that's about the worst example you can give a prospective user like me.
My face is hot just having to type this.
It has value even if every person doesn't individually read every line.
Maybe your face is getting hot from trying to shame somebody into giving their work away for free?
Yes, but you are also literally going to multiply several billions numbers together and pray for a random outcome.
I don't know, maybe some middle ground would be feasible - like have the server-side part open-source, because that's the only part where anything harmful might be (and it's also fairly thin BTW).
But no matter what, the project needs to be profitable for me, sorry to say that, but let's be a bit realistic, you wouldn't work for free either.
Yes, and a big part of establishing trust is sending the right signals. The risk vs reward differential is way higher for open-source compared to closed source, so all other things being equal I would trust open-source more. Note that all other things are actually not equal in this case, e.g. your closest competition is open-source: Ollama, GPT4All, Llamero.
This probably isn't the best justification to use on tech forums. Odds are higher here and in similar forums that some of audience do indeed read the source code, do static analysis on them, check the libraries for CVEs, etc.
With that said, i'd never pay $5/m these days for something unless is had very good value to me and obvious ongoing costs i'm using (servers/etc) for the dev/company. Those costs would need to be clear though, not just "we chose to run some servers so that we'd have features which justified subscriptions"
I would however, pay for upgrades - ie the licensing model employed by JetBrains/etc. Which can come out to $5/m (or yearly purchases, etc), or w/e, but most importantly the software keeps working if you decide to cancel.
edit: Also i'm on Linux. Looks like i can't buy it anyway haha
But that makes OSS more complicated - people usually don't pay for something they can clone & build themselves.
BTW: Linux and Windows will definitely happen, I just didn't have time yet. Sorry, it's a lot of work and time so I had to pick macos first, and even that alone is a lot of testing on different configurations.
Ideally your app would be all FOSS components and i pay for the glue to tie it all together, but i've seen that become too easy to knock-off. Depends on the app of course. Still, FOSSing parts can do a fair amount of good will to the FOSS crowd, imo.
I’ve got a similar app, with the same goals and eventually same supported platforms. Slight difference is that it’s open source:
https://github.com/BrutalCoding/shady.ai
Thank you for this!
I still salute products like Avapls. Hope you succeed @cztomsik. It’s a good initiative to make offline models accessible.
Jokes aside, thanks for building this! It looks swell and I really enjoy your sense of humor on your readme as well. Looking forward to poking around tonight!
Do you have any interest in supporting older versions of OS X?
EDIT: it's too old, sorry.
Notes:
- I was missing file sizes on the models list, and even in the installer window there's only a percentage indicator. Blurring the window while installing also stops me from visually exploring the app while waiting for the install to finish (maybe just a lil' bit of blur is enough to show that the rest of the window is inactive?).
- Scrolling up should stop auto-scroll in the chat, I couldn't even reach the Stop generation button while it was generating. Maybe put that button in a static location.
- Maybe an uninstall button for models is appropriate.
> Scrolling up should stop auto-scroll in the chat
This is at the top-of the list for today, check the website/discord/twitter for updates. I want to do both - pin the button at the bottom, and also make sure that if you scroll, it will stay there. Currently, it's one component which contains the button and also scrolls and it was a bad idea.
> Blurring the window while installing
Agreed 100%, I definitely want to improve downloader a lot more but this was good enough for first release.
> Maybe an uninstall button for models is appropriate.
It should be there (in the upper area), I hope the 24h cache in cloudfront is not too much but the link should be pointing to 09-18 version which has Delete button.
I'm currently on an intel mac mini with almost no memory, so I can't even download to test this (I have to close all other programs just so I can build an app in xcode).
I've got a razer blade as well, but obviously mac only doesn't help there.