Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
There is a 5 month old thread [1] on this, but it might be already outdated.
What is the best approach for feeding custom set of documents to LLM and get non-halucinating and decent result in Dec 2023?
UPD: The question is generally about how to "teach" LLM answer questions using your set of documents (not necessarily train your own, so approaches like RAG counts)
[1] https://news.ycombinator.com/item?id=36832572
245 comments
[ 4.5 ms ] story [ 272 ms ] threadYou have to upload your documents to S3, create a “Knowledge Base” then sync your documents into a vector database like OpenSearch or PineCone. You are then good to go via their playground or the AWS API.
I made a video here describing the process, check around 14 minutes in:
https://ensembleanalytics.io/blog/introducing-bedrock-knowle...
Bedrock is a decent product I think. All of the models in one place (apart from the big dogs from OpenAI) and a common API across them.
We are not always makers. Oftentimes we're consumers as well.
I don't want to read documentation and experiment with my phone, I just want it to work out of the box and do what I expect.
This is standard consumer behaviour and you're lying to yourself if you don't think you act like this with some things.
AWS OpenSearch has fairly good integration so you could look up costs for that. It’s not the cheapest AWS service to run and not exactly serverless as you pay by the hour.
This isn't the same as the training data the LLM is trained on. As a result, it doesn't take advantage of the entire document set. So, if you have a billion documents, only 1-3 chunks will be picked for the final answer. When you know a question spans many many documents, the answer is never going to cover that.
You could make a recursive also where you parse all the chunks, generate summaries of those and then pass them to the next chunk sequentially and so on. But you can imagine how expensive and slow that will be. It might still work for you but this is a very lossy approach.
If you have that much data in your vdb, and the user is querying a slice, it just won't do to have them ask something too generic, they need to be asked to be specific.
Ie "what was that thing from last year?" "Sorry, could you be more specific?" "Oh uh the thing to do with the financial reports" "Financial reports, yes, what about them?" "It was something to do with the way we generate them" "Ah yes, the generation of financial reports; last year in December your team leader said on Slack...etc"
I agree it is high and it is not exactly serverless. You pay by the hour.
Bedrock itself is charged based on tokens exchanged.
I think Pinecone is a cheaper database for hobby and small business projects, though I haven't looked into it.
You can reduce a lot by committing to a 6-month contract, but it won't get cheaper than about ~$5,000/mo.
That's prohibitively expensive for small projects.
Fine tuning GPT 3.5 is much cheaper.
Cheshire Cat [0] looks promising. It's a framework for building AI assistants by providing it with documents that it stores as "memories" that can be retrieved later. I'm not sure how well it works yet, but it has an active community on Discord and seems to be developing rapidly.
The main perk over the cloud options is that you can point it at any language model, including fully local—my local install pointed at my local Ollama running Mistral.
[0] https://github.com/cheshire-cat-ai/core
You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG.
Test it out. If it really and truly doesn't work, search for a script that creates question and answer pairs automatically with gpt-4. Then try using that for qLoRA. I have never heard of anyone successfully using that for a private document knowledgebase though. Only for skills like math, reasoning, Python, etc. I think the issue is that you need a LOT of data and it needs to repeat concepts or any facts you need to learn many, many times in different supporting ways.
What absolutely does not work is trying to just feed a set of documents into fine tuning. I personally have proven that dozens of times because I had a client who is determined to do it. He has been mislead.
What it will do is learn the patterns that are in those documents.
https://www.microsoft.com/en-us/research/group/dynamics-insi...
LlamaIndex is pretty good.
We originally started out building features with LangChain (loading chains from YAML sounded good—it felt like it would be easy to get non-engineers to help with prompt development) but in practice it’s just way too complicated. Nice idea, but the execution feels lacking.
It also doesn’t help that LangChain is evolving so rapidly. When we first started using it a lot of code samples on the internet couldn’t be copy/pasted because of import paths changing, and at one point we had to bump by ~60 patch versions to get a bug fix, which was painful because it broke all kinds of stuff
Disclaimer: I work at deepset
[1] https://github.com/deepset-ai/haystack
With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed.
I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cases and for situations across the knowledge base.
It did feel like it broke down and got confused at a certain point of complexity of questioning, but I still think it’s already useful as a “copilot” or search engine and surely it will only improve over time.
I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning.
Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?
I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.
Retrieval-Augmented Generation for Large Language Models: A Survey
https://arxiv.org/abs/2312.10997
The photos of this post are also good for a high level look
https://twitter.com/dotey/status/1738400607336120573/photo/2
From the various posts I have seen people claim that phi-2 is a good model to start off from.
If you just want to do embeddings, there are various tutorials to use pgvector for that.
https://kagi.com/search?q=ml+rag
https://www.google.com/search?q=ml+rag
Then you start making more specific queries? How old is he, how tall is he, etc.
And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions might this paragraph answer”, and then turn around and feed those questions and text back into the system.
Is that a 30,000 foot view really of how this works?
You take the users question either embed it directly or augment it for embedding (you can for example use LLM to extract keywords form the question), query the vector db containing the data related to the question and then feed it all of LLM as: here is question form the user and here is some data that might be related to it.
Then you craft a prompt for the model along the lines of "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'".
Then, you take all of your documents and divide them into meaningful chunks, ie by paragraph or something. Then you take these chunks and create embeddings for them. An embedding model is another type (not an llm) that generates vectors for strings of text often based on how similar the words are in _meaning_. Ie if I generate embeddings for the phrase "I have a dog" it might (simplified) be a vector like [0.1,0.2,0.3,0.4]. This vector can be seen as representing a point in a multidimensional space. What an embedding model does with the word meaning is something like if I want to search for "cat" that might embed as a vector [0.42]. Now, say we want to search for the query "which pets do I have" first we generate embeddings for this phrase, the word "pet" might be embedded as [0.41] in the vector. Because it's based on trained meaning, the vectors for "pet" and for "dog" will be close together in our multidimensional space. We can choose how strict we want to be with this search (basically a limit to how close the vectors need to be together in space to count as a match).
Next step is to put this into a vector database, a db designed with vector search operations in mind. We store each chunk, the part of the file it's from and that chunks embedding vector in the database.
Then, when the LLM is queried, say "which pets do I have?", we first generate embeddings for the query, then we use the embedding vector to query our database for things that match close enough in space to be relevant but loose enough that we get "connected" words. This gives us a bunch of our chunks ranked by how close that chunks vector is to our query vector in the multidimensional space. We can then take the n highest ranked chunks, concatenate their original text and prepend this to our original LLM query. The LLM then digests this information and responds in natural language.
So the query sent to the LLM might be something like: "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'
Information:I have a dog,my dog likes steak,my dog's name is Fenrir
User query: which pets do I have?"
All under "information" is passed in from the chunked text returned from the vector db. And the response from that LLM query would ofc be something like "You have a dog, its name is Fenrir and it likes steak."
(Seems like this is what reinforcement training is, but I am just not sure? Everything seems to mush together when talking about gpts logic)
For example you can have a table of "knowledge" as I described earlier, but you can just add easily have a table of the conversation history, or have both.
In fact it's quite popular afaik to store the conversation this way because then if you query on a topic you've queried before, even if the conversation history has gone behind the size of the context, it can still retrieve that history. So yes, what you describe is a good idea/would work/is being done.
It really all comes down to the non model logic/regular programming of how your vector db is queried and how you mix those query results in with the user's query to the LLM.
For example you could embed their query as I described, then search the conversation history + general information storage in the vector db and mix the results. You can even feed it back into itself in a multi step process a la "agents" where your "thought process" takes the user query and breaks it down further by querying the LLM with a different prompt; instead of "you are a helpful assistant" it can be "you have x categories of information in the database, given query {query} specify what data to be extracted for further processing" obv that's a fake general idea prompt but I hope you understand.
Well there's technically no model training involved here but I guess you consider the corpus of conversation data a kind of training, and yeah that would be RLHF based which LLMs learn pretty heavily on afaik (I've not fine tuned my own yet).
You can fine tune models to be better at certain things or respond in certain ways, this is usually done via a kind of reinforcement learning (with human feedback...idk why it's called this, any human feedback is surely just supervised learning right?) this is useful for example, to take a model trained on all kinds of text from everywhere, then fine tune it on text from scifi novels, to make it particularly good at writing scifi fiction.
A fine tune I would say is more the "personality" of the underlying LLM. Saying this, you can ask an LLM to play a character, but the underlying "personality" of the LLM is still manufacturing said character.
Vector databases are more for knowledge store, as if your LLM personality had a table full off open books in front of them; world atlases, a notebook of the conversation you've been having, etc.
Eg, personality: LLM fine tune on all David Attenborough narration = personality like a biologist/natural historian
Knowledge base = chunks of text from scientific papers on chemistry + chunks of the current conversation
Which with some clever vector db queries/feeding back into model = bot that talks like Attenboroughish but knows about chemistry.
Tbf the feedback model it's better to use something strict, ie instruct based model, bc your internal thought steps are heavily goal orientated, all of the personality can be added with the final step using your fine tune.
It fascinates me how much variance there is in peoples searching skills.
some people think they are talking to a person when searching e.g 'what is the best way that i can {action}' I think the number one trick is to forget grammar and other language niceties and just enter concepts e.g. 'clean car best'
The same term was the name of the device + something about the power source.
The result from the client network - my phone/client computer nothing related to the search for 4-5 pages.
Same search from work - second result was what I was looking.
So it seems there is a relation with your search history, but somehow connected with the related search history from the same ip/network.
It must be possible to create a fixed set of google searches and rate the location based on the results. So you could physically travel to a Starbucks 20miles away to get the best results for the 'best USB-C dongle reddit'.
Amazon is the worst.
I used "" and + and - for terms to get what I want, and its search engine still gives you the sponsored results and an endless list of matches based on what you might buy instead of what you searched for.
ugh.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
or because you want to charge your client the "ai fee".
or because your indexing is so bad you hide it from your user and blame the llm assistant dept.
It's retrieval augmented generation
I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language SQL querying now.
I'll probably fix some things this week and then either update it or start from scratch. Guided generation, structured extraction, function calling and multi-modal are things I wanted to add and chainlit looks interesting.
We did all we could to steer him toward a correct path of understanding. Sadly we launched a working product but he doesn't understand it and continues to miss represent and miss sell it.
After continuing to give him time and follow up with him (I tend to personally do this with Clients like this), I can tell he is starting to realize his lack of understanding...
Not quite. It does work, albeit likely not optimal.
See https://github.com/bublint/ue5-llama-lora
Things get messy when the number and type of documents increase. Below are the reasons why you may need advanced RAG.
1. Intelligent Data Parser 2. Chunking efficiently 3. Choice of embedding models 4. Query transformation 5. RAG technique 6. Prompt design 7. Feedback loop
Check out my blog on the 27 parameters, considerations and techniques one could follow to build a State-of-the-Art Chatbot.
https://www.lyzr.ai/27-parameters-techniques-considerations-...
So here is the quick guide.
For simpler usecases - inbuilt vector database RAG is sufficient For more complex ones - LlamaIndex or Langchain options are suitable For enterprise grade production use cases - Lyzr's SOTA RAG architecture comes in handy
For the first (fine tuning) follow “AI Jason” on YouTube. He has some great tutorials.
For the second (RAG or similar), fire up a cloud VM with GPUs or use Ollama locally and read through the LlamaIndex docs on how to build a RAG pipeline.
As several other comments in this thread have already indicated: this is almost always the wrong direction. Which is confusing because it's the direction everyone always assumes they should go in at first.
The approaches that does work is surprisingly simple: take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get.
This is known as RAG: Retrieval Augmented Generation. It's a very powerful approach.
We use RAG at my job, but we don’t do any preprocessing on the message from the user, so the results are not always great for us.
Do any of you have experience using a small local model just for extracting keywords from messages which you then use for the retrieval? And then feed the search result and your prompt into OpenAI or whatever as normal.
Also maybe try to include tags or categories when you index and then you can filter on those when doing the vector search. Might get a similar effect from BM25.
Also llamaindex does RAG better than some other solutions.
Embedding and chunking large amounts of documents is expensive though, in both compute and storage.
The other trick I've been planning to explore is using an LLM to turn the user's question into a small number of normal FTS search queries and then run those to try and get context data.
I have also been working on this. I still fail to see why this approach isn't the default frankly. There's little benefit to vector databases.
so is the default to just run the RAG search index on every prompt and if it returns nothing then you get the plain answer from the base model otherwise you get the augmented answer?
ChatGPT is a consumer facing service that wraps the GPT-4 foundational model but at some point will likely wrap gpt-4-turbo.
Signing up for OpenAI API access or Azure OpenAI Services will grant you access to this model (with some rate-limits in place given its a preview model).
ChatGPT-4 is powered by GPT-4 Turbo at this point, so it has a context window that is much larger than 4096 tokens, whether it knows it or not. The ChatGPT application may limit the context size from reaching the full 128k to keep costs down, and it will be using some of the context window for its own purposes, but it's certainly able to maintain context across more than 4096 tokens of conversation.
https://www.anthropic.com/index/claude-2-1-prompting
i.e given a text, always return back a certain set of fields. For some keys here is the possible set of enums etc. One shot prompting does work but curious how others approach this if you have training data on hand.
[0]: https://github.com/outlines-dev/outlines [1]: https://github.com/1rgs/jsonformer
Aside from that, take a look at llama.cpp grammars.
Crude, I know, but it’s compatible with every model. Which is useful if you want to compare the many different models out there.
LLMs are so much better for this than LSTMs now.
I uploaded an AWS study guide, and I'm asking for example questions for my testing. As far as I can see, I can tell no way to determine if it's pulling from the guide, or if it's from GPT's data.
Then create an index about the metadata of each doc. So that you can ask the RAGbot what all it can answer about.
Another way to ensure it stays on-domain is to generate synthetic questions & check for similarity against user queries. There's a whole rabbit hole of query decomposition to avoid straying off topic as well.
Tried this summer, and kinda worked!
Instead you can extract text embeddings from your documents, put them in a vector DB, and then you have a super search. You can convert your search query to an embedding, search the DB and keep the e.g. 10 closest matches.
I'm very happy with its results, even though the system is still young and a little bit janky. You can use it with either GPT API, or your local models through LiteLlm. (I'm running ollama + dolphin-mixtral)
Likely I misunderstood about how RAG works with Azure AI Studio, so sorry in advance
Check it out, specifically steps 3 and 4. As with almost every Microsoft CLI tool and SDK, it's clunky... and you can tell everyone is rushing this AI shit out as fast as they can to stay in the game. But what you want should be doable.
Semantic Kernel is MS's response to LangChain and LlamaIndex - available for .NET, python and Java.
Using their Memory support (using Azure Cognitive Search), it gives a powerful RAG quickstart, which you can combined with Azure Document Intelligence to chunk your source documentation into memories that your foundational models can later use.
(Disclaimer: Its only very recently gone 1.0 and is still likely to undergo API change as the LLM domain itself is still rapidly evolving - I've substantially forked the project for my own needs, but I hope that as it stabilises, I can contribute PRs for some of my more advanced use cases).
I've tried out working with custom documents in two different ways for different types of data:
* Once using LlamaIndex + Chroma[0] to transcribe and then conversationally query video contents (using GPT 3.5 or 4 as the backing LLM).
* Once using GPT Plus, uploading long-form PDFs of my own fiction books to the GPT's knowledge base. I use this to help me remember character names and timelines (not always accurate, so results need to be treated with caution) and help brainstorm story or tech ideas for my world.
Both work for what I'm using them for. I feel like option one is more customizable and easier to tweak for the types of results I would want, if I have concrete requirements about what kind of output I'm looking for. Option two has a lower barrier to entry and is just a little lower effort (no need to run your own app).
For the next iteration, I'd like to try out AWS Bedrock and compare the workflow and results.
[0] https://www.daily.co/blog/search-your-video-content-library-...
Early next year I’m preparing something similar for my team, so I’ll surely look into the useful links/recommendations posted by fellow HNers :-)
Fine tuning does result in degradation of the overall model (https://twitter.com/xaiguydotagi/status/1737082280835703142) and so various RAG techniques may be desirable. As others have mentioned, LlamaIndex is a neat solution to build RAG pipelines: https://docs.llamaindex.ai/en/stable/optimizing/production_r...
I think harpercarrol link is a pretty good one, but it basically just feeds in the documents for completion, which isn't a good approach. The dataset needs to represent how you want to use it.
This one might also be helpful https://www.deeplearning.ai/short-courses/finetuning-large-l...
Honestly surprised how almost everyone is saying to use RAG (on its own). One strong benefit to RAG is the data can change, but has lots of failure modes.
People often use hybrid search (fuzzy or bm25 etc alongside embedding search) which I suppose is still RAG.
But fine-tuning models to be better at RAG is valuable as well, increasing accuracy.
https://ragntune.com/blog/Fine-tuning-an-LLM-to-be-good-at-R...
Ideally, I'd try both. Fine tune on both the documents (create a question / answer dataset with gpt4) and rag instruction fine tune it.