Show HN: Finetune LLaMA-7B on commodity GPUs using your own text (github.com)
I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU.
To prepare the data, simply separate your text with two blank lines.
There's an inference tab, so you can test how the tuned model behaves.
This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio.
Enjoy!
102 comments
[ 0.23 ms ] story [ 184 ms ] threadThat's a good question. I was under the assumption it's linearly proportional, but I can test it out I guess.
By using flash attention, you can get memory requirement down to scale linearly with sequence lengths.
So, I stumbled upon this Simple LLaMA FineTuner project by Aleksey Smolenchuk, claiming to be a beginner-friendly tool for fine-tuning the LLaMA-7B language model using the LoRA method via the PEFT library. It supposedly runs on a regular Colab Tesla T4 instance for smaller datasets and sample lengths.
The so-called "intuitive" UI lets users manage datasets, adjust parameters, and train/evaluate models. However, I can't help but question the actual value of such a tool. Is it just an attempt to dumb down the process for newcomers? Are there any plans to cater to more experienced users?
The guide provided is straightforward, but it feels like a solution in search of a problem. I'm skeptical about the impact this tool will have on NLP fine-tuning.
Actually, you've hit the nail on the head here. I wanted something where I, a complete beginner, can quickly play around with data, parameters, finetune, iterate, without investing too much time.
That's also why I've annotated all the training parameters in the code and UI -- so beginners like me can understand what each slider does to their tuning and to their generation.
And does the training then create windows of ngrams on those chunks? Or what is the input/output?
The reason I ask: If I had question/answer pairs, the question is the input, the answer is the output.
What is the "output" when the input is just a (logically coherent) chunk of text?
It probably won't change much if it's just a single sample. If you put in a large corpus of samples that repeat on the same theme, then the model will be "tuned" to repeat that theme. If you increase the number of epochs, you can overtrain it, meaning that it will just spit out the training data text.
Much cheaper, and simpler than a bare metal machine, data ingress/egress is hard though for Colab you can just mount a Gdrive.
Unfortunately the API for training on M* chips (via MPS) is apparently still extremely buggy, so we have a ways to go before that is fully mainstream. And yes, I know that PyTorch just mainlined their mps support last week too...but from what I've heard the low level interface itself still needs some work. D:
One of those usecases where raw cost doesn't always translate into money saved, at least in my personal experience. :D
It's also nice that you can more easily edit in browser in colab before launching an instance, so you can do development->debugging->training via a single webpage without it breaking the bank or having huge privacy concerns. I think a close second would be Jupyter on vscode w/ a lambda backend but latency + fragmented architecture can add an extra step of complexity (which does matter! D:)
That said, I can definitely take a look at it for scale once I need to go over something like 1-2 hours maybe! :D
Would it also be possible to just train the model from scratch on commodity hardware and how big of a difference in training time would that be?
1. Ask it things. Let it answer.
2. Ask it to find errors in the answer it outputted and for it to correct the answer.
3. Use the original prompt and the corrected output as training data.
This should, with each iteration make the model less and less likely to output statements that are self contradictions or obviously wrong, until the model can no longer spot its own faults.
But if you let an AI's approval be the metric, things turn a lot more fussy and subjective. The goal is not actually "to write a good answer without error" but actually "to write an answer that is approved by the AI". Those are very different goals, and as you keep using it you'll get a bigger and bigger divergence, until eventually the AI is just answering complete garbage nonsense that precisely hits certain sweet spots in the grading AI.
This divergence of the target vs the actual human goal is a pretty interesting problem in AI safety research. I love the example where an AI trained to stay alive as long as possible in Tetris realized that pausing the game was the best strategy.
The correction prompt is very important, it will definitely determine the outcome of the process, a bad correction prompt will obviously lead to a garbage result.
Training in steps with different prompts might be of value. First step might be to fix contradictions, then factual errors if that is an issue. This is an idea that I got when viewing the he output of LLaMA, it often contains contradictions (eg. an example I have seen is "Peter is a boy and he is part of the Gama sorority"). Asking it to fix those types of issues should be a first good step.
But I suspect that this type of training would need to be mixed with original training data. Otherwise the restructuring in the model caused by the new training would most likely garble the rest of the model.
But yeah, you’re going to need an objective metric or human input otherwise the system is going to diverge in strange ways.
This is somewhat similar to how GANs try to learn the density of the underlying data, but here you do not have the underlying data as a reference, if that makes sense. It's sort of like filling a mattress with helium instead of air. Sure, the mattress will be lighter, but that does not mean you will float on it, if that makes any sense at all.
Hope that helps as a cogent answer to this question.
- Self-Instruct: Creates artificial training data on instruction tuning from an untuned base model, from a tiny seed of prompts, and filters out the bad ones before fine-tuning. Manages to approach Instruct-GPT performance with only ~100 human labels. https://arxiv.org/abs/2212.10560
Are you willing to assign a upper limit on this probability and bet for it?
Even recently US copywrite office have asked you too list any parts built with AI as we don't have the laws in place to cover this:
https://www.copyright.gov/ai/
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction: {instruction}
### Response:
or
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction: {instruction}
### Input: {input}
### Response:
[1] https://github.com/cocktailpeanut/dalai
Alpaca/Dalai are finetuned on a dataset that's formatted as this:
So even without pre-prompting in this format it's going to be heavily biased towards performing completions in this format anyways.It's always helpful to finetune on a preformatted prompt depending on what your task is.
Yes, but wasn't the whole point of the recent LLM research to show that you didn't need to fine-tine for a specific task?
Remember that GPT3 is 175 billion parameters so many times bigger than both the above models (and gpt4 is rumoured to be bigger still), which also allows it to be more generalisable.
If GPT3 was trained at 7 billion parameters it might also lose it's language translation capabilities.
this statement that NNs do inference is not entirely correct IMHO.
How does Prolog work? You:
* pass it some predicates * specify rules about what the relationships mean and how various things are computed from the data in predicates * query a variable * answer pops out.
How can GPT do this task? You:
* pass it some predicates (structured machine-readable syntax or natural-language sentences) * specify rules (in natural-language sentences, though it helps to iterate on the wording a bit to make the rules more rigid, and more likely to provide the correct output ~every time) - you don't normally need to specify relationships explicitly because GPT can usually figure it out * include some additional "massaging" wording to get it reproducibly outputting the kind of result you want * query a variable (tell it what to find out/infer from the data) * answer pops out (in human-readable language or structured syntax).
In some ways, they are very much alike. And GPT is much more natural to program with than Prolog.
it is absolutely not fair, established or not, to call non-linear regression inference. may be prognosis, maybe prediction, maybe just approximation. but inference is something that actually does logic, based on facts... and probabilistic theory infers only probability, not facts.
With no real knowledge of LLM and only recently started to understand what LLM terms mean, such as 'model, inference, LLM model, intruction set, fine tuning' whatelse do you think is required to make a took like yours?
This is for education purposes and love to take a jab on creating something like this and and write an inference - such as the dev behind LLaMa inference in Rush.
>I am not familiar with HuggingFace libaries at all, why were they important in your implementaiton? > Gradio - I believe is the UI that allows to plugin different lLM models, I am familiar with text-generation-ui on GitHub that uses Gradio. >LORA I think further fines tines an model -- just like how LLaMa got fine tuned on instruciton set to produce Alpaca model.
This was mee a few weeks ago. I got interested in all this when FlexGen (https://github.com/FMInference/FlexGen) was announced, which allowed to run inference using OPT model on consumer hardware. I'm an avid user of Stable Diffusion, and I wanted to see if I can have an SD equivalent of ChatGPT.
Not understanding the details of hyperparameters or terminology, I basically asked ChatGPT to explain to me what these things are:
I did the same with all the other terms I didn't understand, like "ADAM optimizer", "gradient", etc. I relied on it very heavily and cross-referenced the answers.Looking at other people's code and just tinkering with things on my own really helped.
Through the FlexGen discord I've discovered https://github.com/oobabooga/text-generation-webui where I spent days just playing around with models. This got me into the huggingface ecosystem -- their transformers library is an easy way to get started. I joined a few other discords, like LLaMA Unofficial, RWKV, Eleuther AI, Together, Hivemind and Petals.
I bookmarked a bunch of resources but it's very sporadic. Here are some:
- https://github.com/zphang/minimal-llama/#peft-fine-tuning-wi...
- https://github.com/togethercomputer/OpenChatKit
- https://www.cstroik.com/index.php/2023/02/18/finetuning-an-a...
- https://github.com/huggingface/peft
- https://github.com/kingoflolz/mesh-transformer-jax/blob/mast...
- https://github.com/oobabooga/text-generation-webui
- https://github.com/hizkifw/WebChatRWKVstic
- https://github.com/ggerganov/whisper.cpp
- https://github.com/qwopqwop200/GPTQ-for-LLaMa
- https://github.com/oobabooga/text-generation-webui/issues/14...
- https://github.com/bigscience-workshop/petals
- https://github.com/alpa-projects/alpa
It seems not that hard to include in your prompt “here is some examples, please write like this and follow the style set here”. While that may make every completion request more expensive (more tokens), it also seems like fine tuning these models can also be quite expensive.
I’m curious if there are other trade offs besides cost— maybe quality achieved is better with fine tuning? Very interested to see how it all plays out. On the one hand a massive model like gpt4 can probably be prompted to match any style quite well, albeit costly, vs fine tuning a cheap model may get exactly what you want.
1. train a model
2. fine tune a model
3. create embeddings for a model
4. use few shot prompt examples at inference time
These have decreasing resource need, but also decreasing quality.
For example, the GPT-3 API (not yet the GPT-4 API) has a functionality to send it your own embeddings, for example of your own source code documentation. Then you can query GPT-3 and it "knows" your source code doc and answers specifically with that in mind.
Fine tuning it in just 52k examples (alpaca) makes a night and day difference in usability for instruction following.