Is anyone using self hosted LLM day to day and training it like a new employee
I have this idea to use LLM daily. Train it on my emails / notes / chats . Have it draft replies and I edit them as needed and it learns from that.
Is anyone doing anything like that? I have all of the open source stuff downloaded (models , lollms-webui , promptfoo, etc ) and have been experimenting with the interactive chat stuff . Also txtai to make semantic search.
That all seems pretty mature / progressing nicely . A few more months and I expect a clear reference stack will emerge .
What about the assistant stack ? I invest all these resources to self host and feed in all my data. I want to maximize the ROI.
74 comments
[ 3.1 ms ] story [ 132 ms ] threadWhat I would love to get working at some point is allowing an LLM access to my schedule, notes and goals and then have it help prompt me at appropriate times. "Hey, TJ I noticed you haven't worked out this week, it's sunny today this might be a good time". That sort of thing.
There seem to be good tooling around agents, prompt engineering, RAG etc. The 'glue' around getting the LLM to help figure out when appropriate time(s) to check in with me is the bit I am missing, but that's probably mostly down to me being an artist and only a very very JR hobbyist programmer though.
I think a cron job run say every hour would be good enough. It would just have to collect all the inputs and make a decision about how "valuable" it would be to check in (haven't worked out in 3 days, very valuable), then it's just about connecting it to the right outputs (email, push noti, ...)
The job itself would be cheap and trivial to host on something like lambda
FWIW, it's not really what you're seeking... but there is a plugin that allows you to invoke an LLM from within Obsidian (via Ollama): https://github.com/hinterdupfinger/obsidian-ollama
In short, it allows you to set up prompts to act on selected text directly within a file, e.g. 'Summarize this selection as a markdown formatted list of key points', 'Write a PRD', 'Translate to [Language]', 'Run this as a prompt', etc.
If you work in a Microsoft world, this is what GraphAPI is all about: enabling access to all the things using your personal authentication token. This includes emails, calendar, OneNote, One Drive, and essentially everything. I've been working on making an easy to use OneNote provider with GraphAPI underneath it that I can use to work with the LLM.
https://learn.microsoft.com/en-us/graph/overview
https://learn.microsoft.com/en-us/graph/use-the-api
Like a Topic-based Personality Construct where the model first determines which of its “selves” should answer the question, and then grabs appropriate context given the situation.
Here's a bunch of other related methods,
Summarizing context - https://arxiv.org/abs/2305.14239
continuous finetuning - https://arxiv.org/pdf/2307.02839.pdf
retrieval augmented generation - https://arxiv.org/abs/2005.11401
knowledge graphs - https://arxiv.org/abs/2306.08302
augmenting the network a side network - https://arxiv.org/abs/2306.07174
another long term memory technique - https://arxiv.org/abs/2307.02738
However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC.
Here's a decent overview https://aman.ai/primers/ai/context-length-extension/
Rope - https://arxiv.org/abs/2306.15595
Yarn (based on rope) - https://arxiv.org/pdf/2309.00071.pdf
LongLoRA - https://arxiv.org/pdf/2309.12307.pdf
The bottleneck is quickly going to be inference. Since the current transformer models need the context length ^2, the memory requirements go up very quickly. IIRC a 4090 can _barely_ fit a 4bit 30B model in memory with 4096k context length.
From my understanding some form of RNNs are likely to be the next step for longer context. See RWKV as an example of a decent RNN https://arxiv.org/abs/2305.13048
There is nothing stopping someone from keeping an LLM in online-training mode forever. We don't do that because it's economically infeasible, not because it wouldn't work.
Seems very realistic!
I start to feel like a one eye king under blind people.
I even remember sometimes when I told people specific things.
What works in my experience - structuring the task similar to a human-driven workflow, breaking it down into small steps is needed. Each step could be driven by a small prompt, relevant document fragments (if RAG is used) and condensed essays/tutorials/guides that were written by a powerful LLM (ideally, GPT-4 pre-Turbo).
Using this approach, you could stay well below 8k token limit even on the most demanding tasks.
(Big size contexts are leaky on all LLMs anyway)
For now, you can do all the custom/manual training you want, but gpt4 will almost always outperform it with the right context.
Hopefully that will change in the future. Even then, I don't expect people to want to self-host as in on their own machines. More like custom training, then host either on SAAS or PAAS, or their own on-prem if they have it. Spending the performance of a personal laptop isn't worth the reduction of performance on other tasks. Again, maybe that will change.
On one of the projects I used Chat GPT-4 to write instructions/tutorials that were then executed by local model on a large chunk of data (cleaning up product catalogues).
GPT-4 then reviewed some results and fine-tuned the instructions.
The fun part here - ChatGPT-4 reviewed some user searches, product details and comments of marketing department. Then it generated condensed tutorials on writing good keywords for the products in this system (keywords have to cover unexpected search terms that people would use when searching for a product).
The tutorial was supposed to be for the "junior marketing assistant", but in reality it was fed to Mistral 7B.
The second pass was done similarly. "Hey, ChatGPT, these are some sample keywords that are produced by junior marketing assistant according to your tutorial. Review them and write a short guide on correcting most common problems".
It works nicely.
Other cases of local models that work good - custom embeddings (multi-lingual, mapped to the same vector space), custom TTS, custom STT. These are mostly used for specialised personal assistants.
If your goal is to bake new concepts into the model weights, your only real option is a dataset with that concept being used in a wide variety of contexts.
A more feasible approach I think would be retrial augmented generation. You’d essentially store your conversations in a database and calculate embeddings as you go. This would allow you to later do a natural language search of the database, and insert the most relevant portion of the conversation into your context window.
Sort of how they made chatGPT way more likely to obey requests?
Fine tuning is very unintelligent in the sense that it doesn’t take the context of the training samples into account. It’s a dumb optimizer that’s trying to minimize next token loss. Gradient descent is not beholden to the behaviors you taught in the instruct fine tune step.
There is a high chance that a plain similarity search (dot product or cosine distance) will bring a lot of noise and junk into the request. And high noise/signal ratio in the context tends to lead to hallucinations.
On a high level:
(1) give LLM ability and enough information to "expand" user query into a multiple search phrases. Search engine will use them to find most relevant fragments via a form of embedding search
(2) Get highest ranking document fragments and "show" them to LLM saying: "This are the results that were found in the document database using your search phrases via embedding similarity. Refine the search"
(3) Repeat that a couple of times, then rank final documents and combine them for the final answer synthesis.
I've had good experience hiring support folks and working with them on a shared inbox (we use HelpScout).
If you can free up 5 hours of week of dev time by spending $100/week on a support rep, that's a great rate for dev time.
This assumes you're running a profitable business in the first place. If you're still struggling to find paying customers, then yeah it makes sense to do everything yourself.
In my experience, the quality of support is great!
A year into my business, I brought someone in to do customer support for five hours per week, and it freed up a huge amount of time and mental bandwidth.[0]
Since then, I've expanded support to two non-technical customer service reps and two support engineers. It's rare that I answer a question now because my team can generally answer anything I can.
>If he can free up 5 hours a week for $500 and loses 10 customers because they didn't like the support from someone who spends 5 hours a week on this product they've never used, do you think he wins or loses?
I think this is wildly unlikely in practice.
When you bring on a support rep, you don't just say, "Okay, you're on your own. Do the best that you can." You work to onboard them and make sure they're comfortable answering questions and asking for help when they need it. You also course-correct or jump in if they give someone an incorrect or unhelpful answer.
In my two years of having a support team, I can't think of a single time that a customer said they were dropping us because they were dissatisfied with our support.
Instead, what I've found is that the support team often does a better job than I could have because they have more bandwidth to dig deeply into issues.[1] On about 10% of support requests, customers go out of their way to tell us how satisfied they are with the help we gave them.
Documenting things and onboarding teammates is time-consuming and difficult, but the commenter I was originally responding to said that they've already done that work. They described themselves as "a solo developer answering emails that basically point people to various guides and FAQs I’ve published." If they're already at that point, bringing in a support rep is a no-brainer.
In my experience, the majority of support requests I receive are nontechnical. I sell a physical product, so it's probably higher than a pure SaaS, but it's still about 60% of questions are about billing or they deleted their invoice by mistake or they need to see our tax forms for internal paperwork. That's all stuff that a support rep can handle just as well as I can even if they know nothing about my product.
[0] https://mtlynch.io/solo-developer-year-4/#good-leadership-me...
[1] https://forum.tinypilotkvm.com/-826#post-6
Cost is a huge problem. An immigration lawyer would continuously eat into my personal income. If I don't get state funding to cover it, it makes zero business sense. It would also come with all the liability of a first employee.
A GPT that cites my website, German law and a dozen official websites would be a game changer. It could not give advice, but it could find answers like Phind does. It's just tech, with lower running costs and virtually no obligations (unlike an employee).
I just don't know if the result would be useful and trustworthy enough, and it's very expensive to try.
My conclusion is that I should focus on building a good knowledge base, and when the time is ripe, I can augment it with fancy tech.
I have an extra computer for services like filesharing, samba, nfs, git, firewall, etc, for instance caching the models I'm downloading with a squid proxy, so I can test several UIs downloading the same model over again. Not every UI is offering an easy way to set a single folder to store all gguf files, or it's lacking documentation.
I'm already having a lot of fun. There's people already doing much more than this. I'm more worried about integrating and gluing in a way that will become transparent after the new year, local models or not.
Also how to glue this with obsidian/logseq/neovim/etc in a way that I can use with fewest keystrokes possible, instead just uploading a gigantic context or sensible source code files.
Seriously this is not far from Chat GPT 3.5 in only 6.7GB's and runs on a Macbook Air:
https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF
But yeah current context windows are limiting.
One example was trying to use as a assistant to beat long games, without immediate rewards.
I was trying to log and simultaneously get feedback playing Stardew Valley. gpt-3.5-turbo-1106 basically went along with me and my daughter in a coop session giving nice suggestions, sometimes with huge gaps, but easy enough to ask more about after giving more context.
Mistral 7b and 13B was basically mixing up stardew valley with WoW and Genshin Impact, even giving a lot of context about the day I was, what the npcs answered, or things that I know on how to solve a certain quest. It straight made up non existing towns (stardew valley only has one) etc, etc.
I was running the model on a separate gaming notebook, with nvidia, while playing the game on the one I'm using now.
Mistral hasn't released their own official 13B/30B's yet, but i'm really looking forward to what they can do.
What is crazy is that Ultrafastbert, Speculative, Jacobi, or lookahead decoding could potentially speed up by up to 80x depending on size which could make GPT-4 like models feasible on entry level macs / Phones if similar wizardry is done memory wise.
..Yes im very optimistic after the insane progress over the last months with models like Mistral, Deepseek etc.
An LLM that is specifically trained for Software Development, and to which I feed the code of all my company's repositories. And I keep feeding commits/pull requests.
The idea is that I can query it about architectural issues, code improvements, and other technical aspects at different levels of abstraction (code, architecture, business, etc).
So far, I've played a bit with CodeRabbit and it's "just ok" but it is more of a very small windows to what "could be" than being actually useful.
Since you're using txtai, this article I just wrote yesterday might be helpful: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai
Looks like you've received a lot of great ideas here already though!
1 - https://github.com/neuml/txtai
This works reasonably well with gpt4, but my context is almost always too large for self-hosted models so far.
Currently the stack is gravitates around:
- GPT-4 - either to drive the entire workflow OR generate prompts, plans and guidelines for the local models to execute.
- structured knowledge bases (either derived from existing sources OR curated manually by companies to drive AI assistants).
- embedding search indexes, augmented by full-text search. Usually LLM has access to the search engine and can drive the search as needed, refining the queries if results aren't good enough.
All of that is instrumented with logic to capture user feedback at every single step. This is crucial for the continuous improvement of the model!
Bigger model can use this information once in a while, to improve plans and workflow guidelines to make the overall process more efficient.
AMA, if needed!
Using a bit of boilerplate code (a couple of python files) that I copy to new projects.