Is finetuning still worth it as models advance so fast? Any real-world use cases?
For example, Bloomberg trained a GPT-3.5 class LLM on their financial data last year and soon after GPT-4-8k outperformed it on nearly all finance tasks.
We ended up focusing on having high-quality eval data and an architecture that makes switching to new models easy.
Function calling might be one reason. If your app has a lot of custom functions for interacting with tools, fine tuning may be preferred over using context tokens.
Can you recommend a tutorial/document about fine tuning a small model for selecting the correct functions ? Would be great to have something local that can replace OpenAI functions/tools API.
When it comes to traditional NLP related tasks, LLMs are far below dedicated NLP pipelines like POS tagging and feature tagging. However, fine tuning bridges the gap quite a bit between the two.
It's a narrow domain, but so is most of programming. I think if you're just training a general purpose LLM to be more inclined towards your data -- no, fine tuning is probably not very relevant. But if you're trying to solve a very specific yet fuzzy problem, and LLMs can get you _part_ of the way there, fine tuning is likely your best bet.
Not the OP, but you can take a look at https://spacy.io/usage/spacy-101 to get a sense of what traditional NLP tasks look like. These things can be done much faster than LLMs with appropriate tooling (such as spaCy) and they don’t risk hallucination.
In this comment I was referring to POS tagging and feature extraction.
Another use case for fine tuning we have as well is to reduce a 5-shot prompt that we have to run hundreds of times per request down to a “0-shot” (heavy emphasis on the double quotes.
Run five shot on gpt-4o a couple thousand times, then fine tune on cohere’s command-r or haiku or llama3 8b or whichever small but mighty llm. You can reduce costs by 99%, or somewhere in that ballpark, without really sacrificing quality on 99% of the queries.
I use a programming language that isn’t publicly available and it would be useful to have a model understand it. AI isn’t something I’m super knowledgeable about but fine-tuning sounds like a good way to accomplish that.
Fine tuning can be useful if you need to generate lots of output in a particular format. You can fine-tune on formatted messages, and then the model will generate that automatically. That could save a bunch of tokens explaining the output format in every prompt.
Does this Python package control the LLMs using something other than text? Or is the end result still that that Python package wraps your prompt with additional text containing additional instructions that become part of the prompt itself?
Looks like it actually changes how you do token generation to conform to a given context-free grammar. It's a way to structure how you sample from the model rather than a tweak to the prompt, so it's more efficient and guarantees that the output matches the formal grammar.
The output of the LLM is not just one token, but a statistical distribution across all possible output tokens. The tool you use to generate output will sample from this distribution with various techniques, and you can put constraints on it like not being too repetitive. Some of them support getting very specific about the allowed output format, e.g. https://github.com/ggerganov/llama.cpp/blob/master/grammars/... So even if the LLM says that an invalid token is the most likely next token, the tool will never select it for output. It will only sample from valid tokens.
Quality suffers at very large contexts, and you might want to use your large context for something else (e.g. a long current conversation or a lot of recent data).
I still haven't seen really convincing evidence that fine tuning is useful for the internal corporate data use-case (as opposed to RAG which seems to work really well.)
Finetuning would never add any new knowledge. For internal corporate data, use RAG or train a model from scratch. Finetuning would not help anyone answer a question from that data.
Will share a longer post on this once I finish it. I have tested this multiple times, on bigger models, on custom smaller models, and it does not work.
In a strict sense, finetuning can add new knowledge but for that you need millions of tokens and multiple runs without using LoRA or Peft. For practical purposes, it does not.
I get the sense that you want to do the anti-RAG: take some relatively small corpus of data, train a lora, and then magically have a chatbot that knows your stuff...yeah, that will not work.
But chatbots are only one single use-case. And broadly I think this pattern of LLM-as-store-of-knowledge is a bad one (of course, until ASI, and then it isn't).
That said, you absolutely can impart new knowledge through fine-tuning. Millions of tokens is a rather small hurdle to overcome. And if you're not retraining with original/general data, then your model will become very specialized and possibly overfit...which is not an issue in many instances, and may even be desirable.
I have non-English human data annotated in a format that was designed for a very specific health-related study. LLMs haver never seen these annotations, non-English LLMs are not a top priority for companies, and we can only use offline-first ones for data privacy reasons.
In this scenario fine-tuning a general purpose LM works wonders.
What is your strategy for this? Do you finetune when a new flagship model is made available? You said local first, so I'm guessing you might have finetuned llama. But there are llama fine-tunes available which have better performance than the base model. How do you choose?
Our strategy is to take a well-known, battle-tested model as a base, train it, and then hopefully one day release the fine-tuned model on HuggingFace.
Other than that, fine-tunes don't really matter for us because not many people are rushing to beat the top models on (say) Georgian POS tagging or Urdu sentiment analysis.
As long as the model can turn language into a reasonable vector, we're happy with it.
Fine tuning is generally not worth it over RAG with newer models unless you have a niche data set that's very different from the pretraining data. LoRAs/control vectors are also generally better as they don't induce forgetting or hallucinations, and they're usually "good enough" with RAG.
My sense is that finetuning might still have a role if your use is high-volume and with a narrow/specific goal. For example, we have GPT based summaries of our customer contact calls. It's fairly high volume (a typical bank will handle millions of calls a year).
We're considering finetuning for 2 reasons:
(1) the current system prompt with instructions is getting quite large* and we see more challenges for GPT to stick to the instructions. We could finetune with our historic summaries and a simplified prompt to slightly improve the performance of the summaries / lower token count (performance). The idea would be to then continue improving our system prompt again from that new starting point. Exploration still to be done though.
(2) we might be able to finetune a much smaller model to do what the larger model is currently doing (cost, sustainability)
* The larger instruction prompt is because there are lots of specific needs for the summaries (e.g. how to write the summary, what to exclude (health, etc.), what to include (actions taken), and we e.g. give examples of good vs bad summaries to improve performance.
Now that many open source LLMs are already “useful” it is key to ease the development _around_ them, in particular in a way where users-developers can leverage on “private data” (really, data that is not present in the pre-training of these models).
Even if in the repo it is stated that it is optimized for big models (needing A100/H100), I still feel this can help with smaller models more than large ones.
We can extend “If you build it, they will come” with “If you provide the tools, the will build”
there is also the joy of experimenting a “new and revolutionary” technology. Although yes, with big tech and big salaries there is an economic incentive
The "weights" bit is interesting. HuggingFace's SFTTrainer allows you to train on completions only if you want, but while that makes sense for us humans, LLMs are generally better of learning to predict the entire input. This allows you to get the best of both worlds.
>ps, but for smaller models, such as the 7B a single GPU suffices
If I wanna do this, what GPU would I need?? I have a 3060 TI (laptop one) and i9 with 16 gigs of ram. I don't have AWS quota, or quota in GCP either. have heard of paperspace, but I want to quickly get started with fine tuning Mistral, cause we are planning to use some of their models for an engagement I am working on.
Unless you have zero budget, I would strongly suggest going up to a "gaming" desktop computer. A gaming GPU can put out 300 W of heat no problem. If your laptop GPU did the same it would melt, it's going to be limited to around 100 W. And heat dissipation is directly proportionaly to speed.
Plus in a desktop, you have the ability to upgrade to faster GPU or even go for multiple GPUs.
Caveat: such machines, especially multi-GPU, are loud and produce enough heat to warm up a whole room quickly.
And cloud will likely be cheaper if you believe you will not spend more than 10% of the next couple of years running the GPU at full blast.
>heat dissipation is directly proportionaly to speed.
Power limiting Nvidia GPUs up to around 60% of max TDP can offer a substantial reduction to heat and noise with minimal performance loss, but agreed on everything else.
You can get a hetzner gpu box for 184€ a month. My company has been fine tuning mistal and llama3 on the rtx4000 these have. It is slightly limited in ram at only 20gb but we found dropping the quantisation has helped with larger input token counts.
It will be very interesting to see which tools become the de-facto for each of the common LLM use cases.
The landscape is so fractured, I feel like I haven't even heard of most tools. I encountered Microsoft's olive the other day and it was completely new to me.
64 comments
[ 6.6 ms ] story [ 55.9 ms ] threadFor example, Bloomberg trained a GPT-3.5 class LLM on their financial data last year and soon after GPT-4-8k outperformed it on nearly all finance tasks.
We ended up focusing on having high-quality eval data and an architecture that makes switching to new models easy.
Also I think there’s a bunch of important techniques that just are t shared.
Here is an example of data prepared for fine tuning Llama for function calling...
https://huggingface.co/datasets/mzbac/function-calling-llama...
I'm unaware of any comprehensive guides - we're still in the wild west.
It's a narrow domain, but so is most of programming. I think if you're just training a general purpose LLM to be more inclined towards your data -- no, fine tuning is probably not very relevant. But if you're trying to solve a very specific yet fuzzy problem, and LLMs can get you _part_ of the way there, fine tuning is likely your best bet.
Many of the use cases I've seen for LLMs would actually be better with NLP.
Another use case for fine tuning we have as well is to reduce a 5-shot prompt that we have to run hundreds of times per request down to a “0-shot” (heavy emphasis on the double quotes.
Run five shot on gpt-4o a couple thousand times, then fine tune on cohere’s command-r or haiku or llama3 8b or whichever small but mighty llm. You can reduce costs by 99%, or somewhere in that ballpark, without really sacrificing quality on 99% of the queries.
There's a reference to the paper that describes the method at the bottom of the README: https://arxiv.org/pdf/2307.09702
Internal corporate data GPT4 was never exposed to?
(not mine)
In a strict sense, finetuning can add new knowledge but for that you need millions of tokens and multiple runs without using LoRA or Peft. For practical purposes, it does not.
But chatbots are only one single use-case. And broadly I think this pattern of LLM-as-store-of-knowledge is a bad one (of course, until ASI, and then it isn't).
That said, you absolutely can impart new knowledge through fine-tuning. Millions of tokens is a rather small hurdle to overcome. And if you're not retraining with original/general data, then your model will become very specialized and possibly overfit...which is not an issue in many instances, and may even be desirable.
I have non-English human data annotated in a format that was designed for a very specific health-related study. LLMs haver never seen these annotations, non-English LLMs are not a top priority for companies, and we can only use offline-first ones for data privacy reasons.
In this scenario fine-tuning a general purpose LM works wonders.
Other than that, fine-tunes don't really matter for us because not many people are rushing to beat the top models on (say) Georgian POS tagging or Urdu sentiment analysis.
As long as the model can turn language into a reasonable vector, we're happy with it.
We're considering finetuning for 2 reasons:
(1) the current system prompt with instructions is getting quite large* and we see more challenges for GPT to stick to the instructions. We could finetune with our historic summaries and a simplified prompt to slightly improve the performance of the summaries / lower token count (performance). The idea would be to then continue improving our system prompt again from that new starting point. Exploration still to be done though.
(2) we might be able to finetune a much smaller model to do what the larger model is currently doing (cost, sustainability)
* The larger instruction prompt is because there are lots of specific needs for the summaries (e.g. how to write the summary, what to exclude (health, etc.), what to include (actions taken), and we e.g. give examples of good vs bad summaries to improve performance.
I want it to be fixing knowledge based on my data and fine-tune it to be adjusted to my use case already.
I also want a.feedbackloop and can't start to send more and more context with the payload just because my feedback loop adds / finetunes data
RAG is a far better option for making a model work with your data/information.
Finetuning is better for making it output in a language, style, data format, programming language, etc.
in the current landscape, specialized, smaller systems will still be the most efficient way ahead in the near future.
Even if in the repo it is stated that it is optimized for big models (needing A100/H100), I still feel this can help with smaller models more than large ones.
We can extend “If you build it, they will come” with “If you provide the tools, the will build”
If I wanna do this, what GPU would I need?? I have a 3060 TI (laptop one) and i9 with 16 gigs of ram. I don't have AWS quota, or quota in GCP either. have heard of paperspace, but I want to quickly get started with fine tuning Mistral, cause we are planning to use some of their models for an engagement I am working on.
They list hardware requirements by model and you can select your VRAM and system memory to filter available models.
Plus in a desktop, you have the ability to upgrade to faster GPU or even go for multiple GPUs.
Caveat: such machines, especially multi-GPU, are loud and produce enough heat to warm up a whole room quickly.
And cloud will likely be cheaper if you believe you will not spend more than 10% of the next couple of years running the GPU at full blast.
Power limiting Nvidia GPUs up to around 60% of max TDP can offer a substantial reduction to heat and noise with minimal performance loss, but agreed on everything else.
They also now do hourly rental.
The landscape is so fractured, I feel like I haven't even heard of most tools. I encountered Microsoft's olive the other day and it was completely new to me.