HN Explorer

Show HN: I made an app to use local AI as daily driver (recurse.chat)

637 points by xyc ↗ HN
Hi Hackers,

Excited to share a macOS app I've been working on: https://recurse.chat/ for chatting with local AI. While it's amazing that you can run AI models locally quite easily these days (through llama.cpp / llamafile / ollama / llm CLI etc.), I missed feature complete chat interfaces. Tools like LMStudio are super powerful, but there's a learning curve to it. I'd like to hit a middleground of simplicity and customizability for advanced users.

Here's what separates RecurseChat out from similar apps:

- UX designed for you to use local AI as a daily driver. Zero config setup, supports multi-modal chat, chat with multiple models in the same session, link your own gguf file.

- Import ChatGPT history. This is probably my favorite feature. Import your hundreds of messages, search them and even continuing previous chats using local AI offline.

- Full text search. Search for hundreds of messages and see results instantly.

- Private and capable of working completely offline.

Thanks to the amazing work of @ggerganov on llama.cpp which made this possible. If there is anything that you wish to exist in an ideal local AI app, I'd love to hear about it.

245 comments

[ 3.9 ms ] story [ 269 ms ] thread
CGamesPlay
Possibly a strange question, but do you have plans to add online models to the app? Local models just aren't at the same level, but I would certainly appreciate a consistent chat interface that lets me switch between GPT/Claude/local models.
[deleted]
(comment deleted)
xyc
Not strange at all! It's a very valid ask. The focus is local AI, but GPT-3.5/GPT-4 are actually included in the app (bring your own key), although customization is limited. Planning to expose some more customizability there including API base urls / model names.
castles
https://recurse.chat/faq/#:~:text=We%20support%20Mistral%2C%...
iansinnott
You could try out Prompta [1], which I made for this use case. Initially created to use OpenAI as a desktop app, but can use any compatible API including Ollama if you want local completions.

[1]: https://github.com/iansinnott/prompta

CGamesPlay
This one doesn't seem to support system prompts, which are absolutely essential for getting useful output from LLMs.
derwiki
Can you speak more to this? I get useful output from LLMs all the time, but never use system prompts. What am I missing?
CGamesPlay
Sure, I use one system prompt template to make ChatGPT be more concise. Compare these two: https://sharegpt.com/c/fEZKMIy vs https://sharegpt.com/c/S2lyYON

I use similar ones to get ChatGPT to be more thorough or diligent as well. From my limited experience with local models, this type of system prompting is even more important than with ChatGPT 4.

addandsubtract
Is there a difference in using a system prompt and just pasting the "system prompt" part at the beginning of your message?
CGamesPlay
Haven't tested, but having it built-in is more convenient, and convenience is why I'm using these tools in the first place (as a replacement for StackOverflow, for example).
iansinnott
You can update the system prompt in the settings. Admittedly this is not mentioned in the README, but is customizable.
refulgentis
> the system prompt

There isn't a singular system prompt. It really does matter!

Copy the OpenAI playground, you'll thank yourself later

8n4vidtmkvmk
You use multiple system prompts in a single chat? What for?
iansinnott
Fair point, and it's not implemented that way currently. It's more like "custom instructions" but thanks for pointing that out. I haven't used multiple system prompts in the OpenAI playground either, so I hadn't given it much thought.
a_bonobo
I've run into the same problem with deploying Gemini locally, it does not seem to support System Prompts. I've cheated around this by auto-prepending the system prompt to the user prompt, and then deleting it from the user-displayed prompt again.
longnguyen
Shameless plug: if you need multiple AI Service Provider, give BoltAI[0] a try. It’s native (not Electron), and supports multiple services: OpenAI, Azure OpenAI, OpenRouter, Mistral, Ollama…

It also allows you to interact with LLMs via multiple different interfaces: Chat UI, a context-aware called AI Command and an Inline mode.

[0]: https://boltai.com

raajg
looks promising, but after looking at the website I'm yearning to learn more about it! How does it compare to alternatives? What's the performance like? There isn't enough to push me to stop using ChatGPT and use this instead. Offline is good, but to get users at scale there has to be a compelling reason to shift. I don't think that offline capabilities are going to be enough to get significant number of users.

Another tip, I try out a new chat interface to LLMs almost every week and they're free to use initially. There isn't a compelling reason for me to spend $10 from the get to for a use case that I'm not sure about yet.

FloorEgg
Maybe this isn't for everyone, just the people who place a high value on privacy.
ukuina
But how can I guarantee this app is private?

I'm assuming I cannot block internet access to the app because it needs to verify App Store entitlement.

giblfiz
I mean, ok, then how do you distinguish yourself from LM Studio (Free)
vunderba
If your ultimate goal is privacy, then you should only be looking at open source chat UI front ends:

https://github.com/mckaywrigley/chatbot-ui

https://github.com/oobabooga/text-generation-webui

https://github.com/mudler/LocalAI

And then connecting them to off-line models servers:

- Ollama

- llama.cpp

And you should avoid closed source frontends:

- Recurse

- LM Studio

And closed source models

- ChatGPT

- Gemini

copperx
Are you implying Claude is an open source model?
djfdat
I don't think the list was meant to be exhaustive.
bradnickel
The compelling reason to shift to local/decentralized AI is that all of compute will soon be AI and that means your entire existence will go into it. The question you should ask yourself is do you want everything about you being handled by Sam Altman, Google, Microsoft, etc? Do you want all of your compute dependent on them always being up and do you want to trust their security team with your life? Do you want to still be using closed/centralized/hosted AI when truly open AI surpasses all of them in performance and capability. If you have children or family, do you want them putting their entire lives in the hands of those folks.

Decentralized AI will eventually become p2p and swarmed and then the true power of agents and collaboration will soar via AI.

Anyway, excuse the soap box, but there are zero valid reasons for supporting and paying centralized keepers of AI that rarely share, collaborate or give back to the community that made what they have possible.

gverrilla
> when truly open AI surpasses all of them in performance and capability.

Is this true? I've tried llama last year and it was not very helpful. GPT4 is already full of problems and I have to keep circumventing them, so using something less capable doesn't get me too excited.

tkgally
For an app like this, I would really like a spoken interface. Any possibility of adding text-to-speech and speech-to-text so that users can not only type but also talk with it?
xyc
yes I wish it could talk. It's after other priorities though, but I might try something experimental.
girishso
I will totally pay for something like this if it answers from my local documents, bookmarks, browser history etc.
xyc
Yes it would be the next big focus on this. Personal data connectivity is what I see where local AI would excel - despite model power differences.
chaostheory
Yeah, we’re getting closer to “Her”
_boffin_
Good to know there's a market for that. Currently building out something. Integrating from numerous sources, processing and then utilizing those.

nice.

ssnri
I would even let it have longer processing times for queries to apply against each document in my system, allow it to specialize/train itself on a daily basis…

Use all the resources you want if you save me brainpower

xyc
Agree, there's a non real-time angle to this.
samstave
"give me a summary of the news around this topic each morning for my daily read"

Help me plan for upcoming meetings whereby if I put something in calendar, it will build a little dossier for the event, and include relevant info based on the type of event or meeting, mostly scheduling reminders or prompting you with updates or changes to the event etc.

ssnri
“filter out baby pictures from my family text threads”
Satam
I have doubts about that. Most personal data actually lives in the cloud these days. If you need your Gmail emails, you'll need to use their API which is guarded behind $50k certification fee or so. I think there is a simpler version for personal use, but you still need to get the API key. Who's going to teach their mom about API keys? So I think for a lot of these data sources you'll end up with enterprise AIs integrating them first for a seamless experience.
xyc
I think this is a good take. While there's big enough niche for personal data locally, I'd love if there's a way to solve for email/cloud data requiring API keys.
noduerme
Ideally, though, a sufficiently smart LLM shouldn't need API access. It could navigate to your social media login page, supply your credentials, and scrape what it sees. Better yet, it should just reverse-engineer the API ;)
coev
Why wouldn't you be able to use IMAP over the gmail api? IMAP returns the text and headers of all your emails, which is what you'd want the LLM to ingest anyway.
noduerme
Seconding a sibling question: What $50k API fee? To access your gmail? I've been using gmail since 2008 or so without ever touching their web/app interface or getting an API key. You just use it as an IMAP server.
Satam
To use Google's sensitive APIs in production you have to certify your product and that costs tens of thousands. To be honest, didn't think about imap at first, but it looks like that could be getting tougher soon too https://support.google.com/a/answer/14114704?hl=en. Soon they will require oAuth for imap and with oAuth you'll need the certification: https://developers.google.com/gmail/imap/xoauth2-protocol. If it's for personal use, you might be able to get by with just with some warnings in the login flow but it won't be easy to get oAuth flow setup in the first place.
noduerme
Yeah, Thunderbird integrated oAuth in the last few releases, mainly to keep up with the Gmail and Hotmail requirements. Made it very user-friendly to set up in the GUI right within T-bird. I don't see this being a major obstacle.

I'm not sure I can imagine a scenario in production where Google would, or should, allow API access to individual gmail accounts. What's that for? So you can read all your employees' mail without running your own email server?

akerl_
I'm not sure what you mean.

> You will no longer use a password for access (with the exception of app passwords)

I'm not seeing anywhere that I'd need to pay money to use OAuth via an app like Thunderbird or another email client. That app would either need to support using OAuth to let the user auth and get credentials, or use an app password.

Satam
Right, but Thunderbird had to pay up and set themselves as a middleman to allow this. My point is that local LLMs might not have that many advantages for personal data because most of that data doesn't live locally on your computer, to begin with. I guess an argument could be made that running them locally prevents an AI provider from gobbling up ALL of your data. On the other hand, Google already has most of our my data: emails, youtube, gmail, etc.
samstave
What?

I manage both gmail and protonmail via thunderbird - where I have better search and sort using IMAP.

chb
This. There was a post in HN last week, iirc, referring to just such a solution called ZenFetch (?). I would have adopted it in a heartbeat but they don’t currently have a means of exporting the source data you feed to it (should you elect it as your sole means of bookmarking, etc)
gabev
Hey there,

This is Gabe, the founder of Zenfetch. Thanks for sharing. We're putting together an export option where you can download all your saved data as a CSV and should get that out by end of week.

samstave
Seems like this would be a good tool to build lessons on - if you could share a "class" and export a link for others to then copy the class, and expand on the lesson/class/topic into their own AI. but as a separate "class" and not fully integrated to my regular history blob?

I want the ability to search all my downloaded files and organize them based on context within. Have it create a category table, and allow me to "put all pics of my cat in this folder, and upload them to a gallery on imgur."

gabev
We're working on the ability to share folders of your knowledge so that others can search/chat across them.

We've been thinking of this as a "subscription" to the creator's folder. Similar to how you might subscribe to a Spotify playlist

toomuchtodo
Consider using tar files for this. Lots of tooling (versioning, hashing, storage) around this already, and docker layers comes to mind.
scottrblock
plus one, I would love to configure a folder of markdown/txt(+ eventually images and pdfs) files that this can have access to. Ideally it could RAG over them in a sensible way. Would love to help support this!
xyc
Thank you! I'd love to learn more about your use cases. Would you mind sending an email to feedback@recurse.chat or DM me on https://x.com/chxy to get the conversation started?
vunderba
There are already several RAG chat open source solutions available. Two that immediately come to mind are:

Danswer

https://github.com/danswer-ai/danswer

Khoj

https://github.com/khoj-ai/khoj

wkat4242
Stupid question but what does RAG stand for?
onehp
Retrieval augmented generation. In short you use an LLM to classify your documents (or chunks from them) up front. Then when you want to ask the LLM a question you pull the most relevant ones back to feed it as additional context.
danielovichdk
I dont get it. To my understanding it takes huge amounts of data to build any any form of RAG. Simply because it enlarges the statistical model you later prompt. If the model is not big enough how would you expect it to answer you in a non qualifying matter ? It simply can't.

So I don't really buy it and I have yet to see it work better than any rdbms search index.

Tell me I am wrong, I would like to see a local model based on my own docs being able to answer me quality answers based on quality prompts.

tveita
RAG doesn't require much data or involve any training, it is a fancy name for "automatically paste some relevant context into the prompt"

Basically if you have a database of three emails and ask when Biff wanted to meet for lunch, a RAG system would select the most relevant email based on any kind of search - embeddings are most fashionable, and create a prompt like

"""Given this document: <your email>, answer the question "When does Biff want to meet for lunch?"""

loudmax
That's not how RAG works. What you're describing is something closer to prompt optimization.

Sibling comment from discordance has a more accurate description of RAG. There's a longer description from Nvidia here: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge...

tveita
Right, you read something nebulous about how "the LLM combines the retrieved words and its own response to the query into a final answer it presents to the user", and you think there is some magic going on, and then you click one link deeper and read at https://ai.meta.com/blog/retrieval-augmented-generation-stre... :

> Given the prompt “When did the first mammal appear on Earth?” for instance, RAG might surface documents for “Mammal,” “History of Earth,” and “Evolution of Mammals.” These supporting documents are then concatenated as context with the original input and fed to the [...] model

Finding the relevant context to put in the prompt is a search problem, nearest neighbour search on embeddings is one basic way to do it but the singular focus on "vector databases" is a bit of hype phenomenon IMO - a real world product should factor a lot more than just pure textual content into the relevancy score. Or is your personal AI assistant going to treat emails from yesterday as equally relevant as emails from a year ago?

discordance
RAG:

1. First you create embeddings from your documents

2. Store that in a vector db

3. Ask what the user wants and do a search in the vector db (cosine similarity etc)

4. Feed the relevant search results to your LLM and do the usual LLM stuff with the returned embeddings and chunks of the documents

bigfudge
Although RAG is often implemented via vector databases to find 'relevant' content, I'm not sure that's a necessary component. I've been doing what I call RAG by finding 'relevant' content for the current prompt context via a number of different algorithms that don't use vectors.

Would you define RAG only as 'prompt optimisation that involves embeddings'?

eevmanu
Sure thing, your RAG approach sounds intriguing, especially since you're sidestepping vector databases. But doesn't the input context length cap affect it? (chatgpt plus at 32K [0] or gpt4 via open ai at 128K [1]) Seems like those cases would be pretty rare though.

[0]: https://openai.com/chatgpt/pricing#:~:text=8K-,32K,-32K

[1]: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...

bigfudge
Yes, context window is a limiting factor, but that's true however you identify the content to augment generation.
Grimblewald
You're misunderstanding. Imagine your query is matched against chunks of text from the database, where the relevance of information is evaluated for the window each time it slides. Then collecting the n most relevant chunks, these are included in the prompt so then the llm can provide answers from source documents verbatim. This is useful for cases where precise and exact answers are needed. For example, searching the docs for some package for the right api to call. You don't mant a name that's close to right it has to be correct to the character.
wkat4242
Ahh ok I see. It's basically what MS CoPilot 365 does too, with its "Grounding" step.
spiderfarmer
Next version of MacOS will probably have that.
tethys
As long as you use Safari for browsing, Notes for note taking, iCloud for mail …
Fnoord
I use paperless-ngx for that.
rbtprograms
Looks great! Does it support different sized models, i.e. can I run llama 70B and 7B, and is there a way to specify which model to chat with? Are there plans to allow users to ingest their own models through this UI?
xyc
If you have a gguf file you can link it. For ingesting new models - I'm thinking about adding some CRUD UIs to it, but I'd like to keep a very small set of default models.
3abiton
How different is this compared to Jan.ai for example?
xyc
as i understand jan.ai is more focused on enterprise / platform, while I'd see where recursechat would go is more like "obsidian.md" but as your personal AI.
gexla
Obsidian has add-ons which do much of this.
internetter
People are treating Obsidian like it's the next Emacs
rexreed
What are the MacOS and hardware requirements? How does it perform on a slightly older model, lower powered Mac? I wish I could test this to see how it would perform, and while it's only $10, I don't want to spend that just to realize it won't work on my older, underpowered Mac mini.
xyc
Good question, I'll put some system requirements on the website. It only supports mac with Apple Silicon now, if that's helpful.
pantulis
Instant buy, great work and the price point is exactly right. Good luck!
xyc
Appreciate your support. Thank you so much!
giblfiz
So there are a few questions that leap out at me:

  * What are you using for image generation? Is that local as well (stable diffusion?) Does it have integrated prompt generation? 


  * You mention the ability to import ChatGPT history, are you able to import other documents?

  * How many "agent" style capacities does it have? Can it search the web? use other APIs? Prompt itself? 

  * Does it have a plugin framework? you mention that it is "customizable" but that can mean almost anything. 

  * What is the license? what assurances do users have that their usage is private? I mean, we all know how many "local" apps exfiltrate a ton of data.
hanniabu
> What are you using for image generation?

It doesn't look like it supports image generation unfortunately. If it did then I would definitely adopt this as my daily driver.

android521
how big is the local model? what is the Mac spec requirement? I don't want to download and find out it won't work in my computer. It seems like the first question everyone would ask and should be addressed on the website.
visarga
It uses ollama which is based on llama.cpp, and adds a model library with dozens of models in all quant sizes.
xyc
no this doesn't use ollama, just based on llama.cpp.
xyc
Appreciate the feedback! It works on mac with Apple Silicon only. I'll put some system requirements on the website.
pentagrama
Sadly I can't try this because I'm on Windows or Linux.

Was testing apps like this if anyone is interested:

Best / Easy to use:

- https://lmstudio.ai

- https://msty.app

- https://jan.ai

More complex / Unpolished UI:

- https://gpt4all.io

- https://pinokio.computer

- https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generat...

- https://github.com/LostRuins/koboldcpp

Misc:

- https://faraday.dev (AI Characters):

No UI / Command line (not for me):

- https://ollama.com

- https://privategpt.dev

- https://serge.chat

- https://github.com/Mozilla-Ocho/llamafile

Pending to check:

- https://recurse.chat

Feel free to recommend more!

chown
I am the author of Msty app mentioned here. So humbled to see an app that is just about a month old that I mostly wrote for my wife and some friends to begin with (who got overwhelmed with everything that was going in LLM world), on the top of your list. Thank you!
Datagenerator
Looks interesting, but can't see what it is doing. Any link to the source code?
petemir
If you need help for testing the Linux version let me know, I’d be happy to help
chown
I was actually looking for one! What's the best way to reach you? Mind jumping on our Discord so that I can share the installer with you soon?
crooked-v
One bit of feedback: there's nowhere to put system messages. These can be much more influential than user prompts when it comes to shaping the tone and style of the response.
chown
That's on the top of our list. It got pushed back because we want to support creating a character/profile (basically select a model and apply some defaults including a system prompt). But I feel like that was a mistake tomwait for it. Regardless, it is getting added in the next release (the one after something that is dropping in a day or 2, which is a big release in itself)
hanniabu
1) What are the mac system requirements? Does it need a specific OS version?

2) If you're privacy first, many would feel a lot more comfortable if this was released as an app in the app store so it will be sandboxed. This is important because it's not open source so we have no idea what is happening in the background. Alternatively open source it, which many here have requested.

lolpanda
Oh thanks! didn't know there are quite a few ChatGPT local alternatives. I was wondering what users they are targeting. Engineers or average users? I guess average users will likely choose ChatGPT and Perplexity over local apps for more recent knowledge of the world.
chown
Hi. I'm the author of Msty app, 2nd on the list above. You are right about average users likely choosing ChatGPT over local models. My wife was the first and the biggest user of my app. A software engineer by profession and training but she likes to not worry about LLM world and just to use it as a tool that makes you more productive. As soon as she took Msty for a ride, I realized that some users, despite their background, care about online models. This actually led me adding support for online models right away. However, she really likes to make use of the parallel chat feature and uses both Mistral and ChatGPT models to give same prompt and then compare the output and choose the best answer (or sometimes make a hybrid choice). She says that being able to compare multiple outputs like that is a tremendously helpful. But that's the extent of local LLMs for her. So far my effort has been to target a bit higher than the average users while making it approachable for more advanced users as well.
Gunnerhead
I’m looking for a ChatGPT client alternative, i.e. I can use my own OpenAI API key in some other client.

Offline isn’t important for me, only that $20 is a lot of money, when I’d wager most months my usage is a lot less. However, I’d still want access to completion, DALL-E, etc.

Would Msty be a good option for me?

chown
Give it a try and see how you feel. "Yes, it will" be a dishonest answer to be completely honest at least at this point. The app has been out for just about a month and I am still working in it. I would love a user like you to give it a try and give me some feedback (please). I am very active on our Discord if you want to get in touch (just mention your HN username and I will wace).
Gunnerhead
Thank you so much, I’m excited to give this a try in the next few days.
AriedK
Looks great, though the fact that you have to ignore your anti-virus warning during installation, and the fact that it phones home (to insights.msty.app) directly after launch despite the line in the FAQ on not collecting any data makes me a little skittish.
stlhood
Just FYI, llamafile includes a web-based chat UI. It fires up automatically.
joshmarinacci
Do any of these let you dump in a bunch of your own documents to use as a corpus and then query and summarize them ?
windexh8er
Yes, GPT4All has RAG-like features. Basically you configure some directories and then have it load docs from whatever folders you have enabled for the model you're currently using. I haven't used it a ton, but I have used it to review long documents and it's worked well depending on the model.
chown
Author of Msty here. Not yet but I am already working on the design for it to be added in very near future. I am happy to chat more with you to understand your needs and what you are looking in such apps. Please hop on the Discord if you don't mind :)
hanniabu
Some of my usecases would be summarizing a PDF report, analyzing json/csv data, upload a dev project to write a function or feature or build a UI, rename image files, categorize images, etc
8n4vidtmkvmk
The new one straight from Nvidia does I believe.
Datagenerator
Open-WebUI has support for doing that, it works using #tags for each document so you can ask questions about multiple specific documents.
[deleted]
(comment deleted)
theolivenbaum
We just added local LLM support to our curiosity.ai app too - if anyone wants to try we're looking for feedback there!
hmdai
Try this one: https://uneven-macaw-bef2.hiku.app/app/

It loads the LLM in the browser, using webgpu, so it works offline after the first load, it's also PWA you can install. It should work on chrome > 113 on desktop and chrome > 121 on mobile.

wanderingmind
lmstudio is using a dark pattern I really hate. Don't have a Github logo in your webpage if your software is not source available. It just takes to Github to some random config repos they have. This is poor choice in my opinion.
Hugsun
We call that stolen valor.
woadwarrior01
Since I couldn't find it in your list, I'd like to plug my own macOS (and iOS) app: Private LLM. Unlike almost every other app in the space, it isn't based on llama.cpp (we use mlc-llm) or naive RTN quantized models (we use OmniQuant). Also, the app has deep integrations with macOS and iOS (Shortcuts, Siri, macOS Services, etc).

Incidentally, it currently runs Mixtral 8x7B Instruct[2] and Mistral[3] models faster than any other macOS app. The comparison videos are with Ollama, but it generalizes well to almost every other macOS app that I've seen uses llama.cpp for inference. :)

nb: Mixtral 8x7B Instruct requires an Apple Silicon Mac with at least 32GB of RAM.

[1]: https://privatellm.app/

[2]: https://www.youtube.com/watch?v=CdbxM3rkxtc

[3]: https://www.youtube.com/watch?v=UIKOjE9NJU4

sigmoid10
What's the performance like in tokens/s?
woadwarrior01
You can see ms/token in a tiny font on the top of the screen, once the text generation completes in both the videos I'd linked to. Performance will vary by machine. On my 64GB M2 Mac Studio Max, I get ~47 tokens/s (21.06ms/token) with Mistral Instruct v0.2 and ~33 tokens/s (30.14ms/token) with Mixtral Instruct v0.1.
castles
Interesting! What's the prompt eval processing speed like compared to llama.cpp and kin?
woadwarrior01
I haven't run any specific low level benchmarks, lately. But chunked prefilling and tvm auto-tuned Metal kernels from mlc-llm seemed to make a big differenced, the last time I checked. Also, compared to stock mlc-llm, I use a newer version of metal (3.0) and have a few modifications to make models have a slightly smaller memory and disk footprint, also slightly faster execution. Because unlike the mlc-llm folks, I only care about compatibility with Apple platforms. They support so much more than that in their upstream project.
castles
thanks, I'll give it a crack
iknowstuff
MacGPT is way handy because of a global keyboard shortcut which opens a spotlight-like prompt. I would love to have a local equivalent
greggsy
Khoj was one of the first 'low-touch' solutions out there I think. It's ok, but still under active development, like all of them really.

https://khoj.dev/

quickthrower2
Thanks for the list. Tried Jan just now as it is both easy and open source. It is a bit buggy I think but the concept is ace. The quick install, tells you which models work on your machine, one click download and then a chatgpt style interface. Mistral 7B running on my low spec laptop at 6 token/s making some damn sense is amazing. The bugs are at the inference time. Could be hardware issues though, not sure. YMMV
smnscu
Nice, adding these to my list. Here's a list that I put together, it has active GitHub projects for LLM UIs, ordered by stars:

- https://github.com/nomic-ai/gpt4all

- https://github.com/imartinez/privateGPT

- https://github.com/oobabooga/text-generation-webui

- https://github.com/FlowiseAI/Flowise

- https://github.com/lobehub/lobe-chat

- https://github.com/PromtEngineer/localGPT

- https://github.com/h2oai/h2ogpt

- https://github.com/huggingface/chat-ui

- https://github.com/SillyTavern/SillyTavern

- https://github.com/ollama-webui/ollama-webui

- https://github.com/Chainlit/chainlit

- https://github.com/LostRuins/koboldcpp

- https://github.com/ParisNeo/lollms-webui/

[deleted]
(comment deleted)
bradnickel
Love this! Just purchased. I am constantly harping on decentralized AI and love seeing power in simplicity.

Are you on Twitter, Threads, Farcast? Would like to tag you when I add you to my decentralized AI threads.

bradnickel
Found your Twitter account in a previous post. Just tagged you.
xyc
Awesome, thanks for the tag!
xyc
Wow, I did not expect at all this will end up on the front page. Thank you for all the enthusiasm, I'll try to get to more questions later today but if there's something I missed my X/twitter DM is open: https://x.com/chxy
castles
It seems "local" is all you need :)
sen
This is awesome. I currently use Ollama with OpenWebUI but am a big fan of native apps so this is right up my alley.
xyc
Thank you!
toomuchtodo
Hey! This is awesome! How hard would it be to plug it into something like Raindrop.io (bookmark manager) to train on all bookmarks collected?
xyc
haven't tried Raindrop.io, looks neat! Saw some other posts mentioning bookmarks as well. I'll keep this in thought, but will have to try it out first to find out.
[deleted]
(comment deleted)
cooper_ganglia
I read the website for 30 seconds and instantly bought it.

It's clean, easy to use, and works really well! Easy local server hosting was cool, too. I've used the other LLM apps, and this feels like those, but simplified. It just feels good to use. I like it a lot!

I'm gonna test drive it for a while, and if I keep using it regularly, I'll definitely be sending in some feedback. Other users have made a lot of really great recommendations already, I'm excited to see how this evolves!

xyc
Thanks so much for the kind words and giving it a spin!

Feel free to send feedback, issues, feature suggestion as you use it more, I'm all ears. My twitter DM is also open: https://x.com/chxy.

devinprater
There's another one someone made for blind users like themselves and me, called Vollama (they use a mac, so VoiceOver + Llama). It's really good. I haven't tested many others for accessibility, but it has RAG and uses Ollama as backend, so works very well for me.

https://github.com/chigkim/VOLlama/

chown
It's very nice that there exists something like that. I am an author of one of the similar apps [1] someone listed in a different thread. I was hoping I could get in touch with someone like you who could give me some feedback on how to make my app more accessible for users like you. I really want to it be an "LLM for all" kind of app but despite my best efforts and intention, I suck at it. Any chance of getting in touch with you and get some feedback? Only if you want and have time, no pressure at all.

[1] https://msty.app

devinprater
Sure, I'll probably join the discord tomorrow morning, but a few notes:

* For apps like this, using live regions to speak updates may be helpful. either that or change the buttons, like from "download local AI" to "configuring." Maybe a live region would be best for that one since sighted people would probably be looking near the bottom for the status bar, but anyway... * Using live regions for chats is pretty important, because otherwise we don't know when a message is ready to read, and it makes reading those messages much simpler. The user types the message, presses Enter, and the screen reader reads the message to them. So, making a live region, and then sending the finished message, or a finished part of a message, to that live region would be really helpful. * Now on to the UI. At the top, we have "index /text-chat-sessions". I guess that should just say "chats"? Below that, we have a list, with a button saying the same thing. After that list with one item, is a button that says "index /local-ai". That should probably just be "local AI". Afterwards, there is "index /settings", which should just be "settings." Then, there is an unlabeled button. I'm guessing this is styled to look like a menu bar, across the top of the window, so it'd be the item on the right side. Now, there's a button below that that says "New Chat^N". I, being a technical user, am pretty sure the "^N" means "Control + N", but almost no one else knows that. So, maybe change that text label. Between that and the Recent Chats menu button are two unlabeled buttons. I'm not sure why a region landmark was used for the recent chats list, but after the chat name "hello" in this case, where I can rename the chat, there is an unlabeled button. The button after the model chooser is unlabeled as well. After the user input in the conversation, there are three unlabeled buttons. After the response, there is a menu button with (oh, that's cool) items to transform the response into bullets, a table, ETC. but that menu button was unlabeled so I had to open it to see what's inside. After that, all other buttons, like for adding instructions to refine this message, are also unlabeled.

So, live regions for speaking chat messages and state changes like "loading" or "ready" or whatever (keep them short), and label controls, and you should be good to go.

Live regions: https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...

chown
Wow! This is already very helpful and was the kind of feedback I was looking for. Thank you!
indit
Hi, I just use msty. Could it use an already downloaded gguf file?
chown
Not right now but that’s something we plan to support soon. Supporting Ollama downloaded models is getting released either today or tomorrow, gguf support might go into the next release. Would love to chat with you to learn more about your use case. Mind saying hi on our Discord?
karolist
Hey. I'm sorry about your condition. I feel I'm approaching blindness eventually, this is very random, but perhaps you could share any resources I could learn to prepare for this so I could continue using the web when/if it happens.
SkepticMystic
I've found great utility with `llm` https://llm.datasette.io, a CLI to interact with LLMs. It has plugins for remote and local models.
xyc
Good to know. I've learned lots of things from Simon Willison's blog (datasette's author), so can't imagine llm being unuseful.
geniium
I am very glad to see that kind of app. Well done!
rkuodys
Honest question - can it be used for programming? Or anyone maybe can recommend local-first development LLM which would take in all project (Python / Angular) and write code based on full repo, not only the active window as with Copilot or Jetbrains AI
arzke
Have you tried using Copilot's @workspace command in the chat?
_ink_
Check out the continue dev plugin (available for VS Code and Jetbrains). You can attach it to OpenAI or local models and it can consider files in your codebase. It has a @Codebase keyword, but so far I get better results in specifically pointing to the needed files.
surrTurr
any plans on supporting ollama integration?
code51
Thank you for the work.

Please take this in a nice way: I can't see why I would use this over ChatbotUI+Ollama https://github.com/mckaywrigley/chatbot-ui

Seem the only advantage is having it as MacOS native app and only real distinction is maybe fast import and search - I've yet to try that though.

ChatbotUI (and other similar stuff) are cross-platform, customizable, private, debuggable. I'm easily able to see what it's trying to do.

vood
Thanks for sharing ChatbotUI. While I'm not an author, I use it extensively and contribute to it. Thanks to the permissive license, I could offer ChatbotUI as a hosted solution with our API keys. https://labs.writingmate.ai.
kvakkefly
Dangerously close to the old famous Dropbox comment to the original ShowHN post
911e
Not a bit of open code while I'm 100% sure they use some that require it. If you"re using AI + Your data without insight on how it's used you're a fool. 2 cents