This isnt specifically Colab or HF, but have you checked out any of the community Runpod templates? Theres a few out there that give you a mostly turnkey way to deploy models and test via Oobabooga, koboldai, or similar.
I use the one-click UI from TheBloke pretty frequently for inference testing, and I know theres some newer ones that also give you fine-tuning capabilities as well.
You use these templates on the runpod platform itself. Theres no free tier equivalent like you have with Colab/HF, but currently you can rent an RTX 4090 for $0.69/hr so its pretty affordable.
In theory yes, since its fundamentally just a Dockerfile, any provider that supports docker is possible. I know HF Spaces have a "docker" sdk option[1], but I personally haven't used them myself.
GPT-J 7B and LLaMA 7B don't look that different in the metrics table, but they are like night and day if finetuned and actually used for question answering, roleplay and such.
If 13B is good... I wonder if this will catch on in the finetuning community.
People care less about the LLaMA license than you'd think, and this is also about the time new models with "improved" architectures (like Falcon) should start popping up.
Pardon me for asking what might be an obvious answer to some, but does increasing the parameters lead to a linear growth of the capability of the neural net, or is it different.
My knowledge of neural nets and AI is just lacking.
There was some recent research that was confirming capabilities improve smoothly as a function of parameter count yes.
Their point was a lot of research that had been showing jumps in performance at certain "breakpoints" for lack of a better word were the result of badly selected metrics versus a case of suddenly emergent behaviour.
The nice thing about that research is it suggests that if you are able to try something on a smaller model it will scale nicely to a bigger model.
This has been known for several years. Especially zero and few-shot task performance scales extremely well with number of parameters. But more recently it was shown that you can actually trade parameters with training data volume and training time as well once you go into the billions of parameters. So while it takes more time to train, you can habe an equally powerful model with much fewer parameters and thus faster inference times.
The paper was fairly recent, https://arxiv.org/abs/2304.15004. It was more thoroughly confirming what was generally agreed upon while debunking other reasons.
The key insight from the abstract, "Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance."
Yup that is another recent and interesting development for sure!
The authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf
Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).
Sorry, I think you've misunderstood, they're saying that's exactly the point. Those weird metrics are what they're debunking, not supporting. Per my snippet from the abstract in my last comment.
Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics.
Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
I think you misunderstood. The authors created the very issue they are "debunking." They took GPT3, slapped on some random metrics and showed that these metrics don't show scaling behaviour correctly, while the original publication of GPT3 actually did it correctly in the first place.
The figures in the paper you linked below are log_10 scale on one axis but not the other. Thus even without quite understanding what all these figures measure, it seems to me that answer the question about "linear growth" is a hard no: double the size will not be twice as good. Or am I misunderstanding how to read these graphs?
Several factors influence performance beyond parameter count, notable ones include: training corpus quality, training flops, and the downstream task.
It depends on how much compute you are spending on training and how big of a model you’re talking about.
There’s a “minimum” tokens/parameter for increasing size to be effective at improving loss/perplexity, so as you go up in parameters you generally will have to broaden your corpus which may lower it’s quality (e.g. tweets/reddit posts vs books/articles).
This effect isn’t as significant at 65B parameters as there is still enough high quality training data but if you’re talking 1T the corpus will (probably, I haven’t tried this/seen this done) by necessity be significantly poorer quality as you will overfit by simply repeating (it’s only beneficial so many times to repeat).
As a general rule, when validation loss/perplexity are the same in two models of different sizes downstream performance seems to be also generally the same (this was briefly explored in the PaLM 2 paper by Google) although it doesn’t correlate perfectly.
Practically speaking, this translates into a bigger model is better for applications we’re generally talking about. It just may not hold infinitely which we’re starting to see evidence of.
It’s definitely not linear though, you can look at some of the OpenLLaMA benchmarks (without getting into the weeds of if current benchmarks are representative) and the accuracy improvements in even 13B is not that significant (noting here all models were trained for the smear number of tokens, so relatively overtraining the smaller models).
There are probably some threshold parameter sizes that make a big difference but it’s still being determined.
Doesn’t llama.cpp resolve all the licensing issues? The models themselves aren’t subject to copyright, so you can use the model weights as long as you haven’t entered into an agreement with Meta about their usage.
Nope. The model itself is GPL-3, legally it doesn't matter if you use Meta implementation or llama.cpp.
Weights being subject to copyright was never tested in court. Businesses have wisely steered away from using leaked weights for commercial purposes. With OpenLLaMA we can expect novel products that incorporate these small but capable LLMs, for example as game NPCs.
Aren't they doing the exact same thing as RedPajama?
How is this not a duplicate effort? Or are they working together with the RedPajama project? If so, why use the OpenLLaMA name?
RedPajama ran on the Summit supercomputer (https://en.wikipedia.org/wiki/Summit_(supercomputer)) NVIDIA V100s/PowerPC chips as part of the INCITE grant which necessitated variations from the LLaMA training parameters.
This led to differences in evals, now they are bringing in more modern chips.
Aside from some tokeniser differences this is a drop in replacement for existing LLaMA that matches performance.
Hey emad, thanks for SD and this! What's the plan if Meta does Apache 2.0 for LLaMA? Just keep going and making the 30b and 65b or build different models?
The tokenizer differences are major as LLMs are sensitive to whitespace handling. If I am reading the github page properly, OpenLLama failed to learn how to model code properly? Code contains many implicit reasoning tasks.
What other differences are there? The page doesn't mention how numbers are handled. These are two major things that impact model reasoning and numeric ability.
Code is main thing, it has some tradeoffs. It tunes well on code though and the code ai team at stability ai are working on stuff.
We can now set and forget runs so will have a better dataset for the next 13b and different tokeniser, this was meant to match as close as possible to be a drop in.
Are there communities where one can go to learn more about fine tuning and running these things? I've found a bunch for diffusion models but haven't had any luck with LLMs.
It's not a community per se but there's a lot of research and discussion going on directly in the llama.cpp repo (https://github.com/ggerganov/llama.cpp) if you're interested in the more technical side of things.
The discord servers for a few of the projects are relatively popular. Most have a help channel you could post in if you have questions. The Discord for KoboldAI has some developers from koboldcpp ,which is the easiest and one of the most bleeding edge way of running these models locally. It builds on llamacpp and allows the use of different front ends among other things like using k quantized models. People also have had success with using something like runpods.
Native fine tuning is still out of consumer reach for the forseeable future, but there's people experimenting with QLORAs. The pipeline is still relatively new though and is a bit involved.
Interesting to see that there’s both Google Cloud and Stability AI mentioned in providing the compute. Did Stability pay the bill for the resources used for training?
If this is 13B and we can assume GPT-4 is 100B+, is this even in the same ballpark as useful?
Being fast at generating "something" (spitting out tokens) is one thing but... if those tokens aren't worth much (aka if the quality of the "answers" are weak), what is it really worth?
The ability to run it on your own private data and train/distill it from the larger models is very useful (see Orca for example: https://arxiv.org/abs/2306.02707).
Most of the valuable data in the world is private data unsuitable to send to GPT4 and other proprietary models.
The GPT4 cognitive leaps and breadth of knowledge seem impossible to replicate with any number of 13B models.
13B x10 might be the only feasible business model for your company, but that alone doesn't make it a superior option. Can you share more details on how the models work together?
I think some of these 13B models may only be pretrained and may not have benefited from RLHF. So it’s not clear how much of the difference is due to parameter count.
Either way, the LLaMa models are quite powerful and now that we have an open, commercially usable version in 3, 7, and 13b sizes is huge. The 65b model I'm assuming is coming soon as well.
If you’re looking to create a ChatGPT replacement, this ain’t it.
If you’re trying to learn about LLMs, how they work, capabilities vs deficiencies, private data use, etc., these are invaluable to get your hands dirty in a private environment.
For me the big difference is that I can run llama models locally which is a different use-case entirely compared to GPT-4. Even if the weights were available, I just don't have the hardware to run something with the parameter counts of GPT-3.5+
They really aren't. 30B models are much smarter, and work fine with 4 bit quantization on a 24GB GPU. On a headless system you can get the full 2048 context size. On a desktop around 1500 I think.
There is the Language Model Evaluation Harness project which evaluates LLMs on over 200 tasks. HuggingFace has a leaderboard tracking performance on a subset of these tasks.
Note that this model can't really be used for most code tasks. The tokenizer removed repeating spaces, so it doesn't have a valid concept of indentation.
I was going to try and be snarky and make a comment about the superior use of tabs, but I just ran samples into the tokenizer and it doesn't recognize tabs either. :-(
Koboldcpp [1], which builds on llamacpp and adds a gui, is a great way to run these models. Most people aren't running these models at full weight, ggml quantization is recommended for cpu+gpu or gptq if you have the gpu vram.
GGML 13b models at 4bit (Q4_0) take somewhere around 9gb of ram and q5_K_M take about 11gb. Gpu offloading support has also been added, I've been using 22 layers on my laptop rtx 2070 max q 8gb vram and CLBlast. I get around ~2-3 tokens per second with 13b models. In my experience, running 13b models is worth the extra time it takes to generate a response compared to 7b models. GPTQ is faster, I think, but I can't fit a quantized 13b model so I don't use it.
TheBloke [2] has been quantizing models and uploading them to HF and will probably upload a quantized version of this online soon. His discord server also has good guides to help you get going, linked in the model card of most of his models.
Edit: There's a bug with the newest Nvidia drivers that causes speed slowdown with large context size. I downgraded and stayed on 531.61 . The theory is that newer drivers change how out of cuda memory management works when trying to avoid OOM errors.
I can actually run the entire Q4_K_S version of this in the gpu with my 3060, it's blazing fast too in this mode (~10 tokens pr second) with the latest llama.cpp, should be the same for kobildcpp too.
I am confident my bottleneck is thermals and hardware related. I had the same speed when comparing koboldcpp and llamacpp a few versions ago. I am running it on a laptop on a full windows 10 install. It's a 10th gen i7 hex, 16g ddr4, and 2070 maxq mobile. In that context, I consider it remarkably fast.
Serious question (since I'm not familiar with AI/ML), what's the point of releasing these "smaller" (5B, 10B, 13B) models, given there are plenty of bigger models now (Falcon 40B, LLaMa 65B)?
A common reason is to reduce cost and latency. Larger models typically require GPUs with more memory (and hence higher costs), plus the time to serve requests is also higher (more matrix multiplications to be done).
As a general principle the larger models are better quality.
However, fine tuned small models can outperform general purpose large models on specific tasks.
There are also many lightweight tasks like basic sentiment analysis where the correctness of small models can be good enough to point of being indistinguishable from large models.
It is very expensive to train these base models so a smaller size is more practical if you aren’t a big company with hundreds of powerful GPUs at hand. Table 15 from LLaMA paper[1] has some insightful figures: it took 135,168 GPU hours to train the 13B version and a bit more than 1M GPU hours for the 65B version. And we are talking about A100 80GB GPUs here (expensive and scarce). Not everyone can afford these kinds of trainings (especially if it takes a few attempts; e.g. if you’ve got a bug in the tokenizer)
Hold on, are you saying I can grab the 13B OpenLLaMa model and train it? I thought all of these models are already pre-trained and represent sort of the end state. Am I completely missing the point?
Falcon 40B is allegedly the closest thing out there. It sure beats the hell out of all the other open source models I've tried, but good luck finding hardware that can run it.
I use locally hosted LLM models for roleplaying almost exclusively, and I've found that the only acceptable models for that use case are 30B or larger (though the jump between a 40B model and a 65B model reaches the point of diminishing returns as far as my experience goes).
What I'd really like to see improve is having larger contexts. The current context size available for llama.cpp-compatible models is 2048 tokens, which given the scenario and character descriptions at the start of the prompt only gives the LLM about 3 paragraphs of memory. It just forgets anything you said more than 3 paragraphs ago, which makes for a pretty miserable longer term RP experience unless you constantly update the summary of what's happened in the story/roleplay so far to be sent with every prompt.
4096 or larger contexts running efficiently (>= 1 T/s) with 12GB/24GB consumer grade GPU layer offloading would be fantastic, and bonus points if we can get 30 or 40B models working with that.
111 comments
[ 3.2 ms ] story [ 156 ms ] threadDoes anyone have any Spaces/Colab notebooks/etc to try this out on?
Thanks!
https://github.com/LostRuins/koboldcpp
Its a llama.cpp wrapper descended from the roleplaying community, but works fine (and performantly) for questioning and such.
You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run
Not sure about colab at the moment.
I use the one-click UI from TheBloke pretty frequently for inference testing, and I know theres some newer ones that also give you fine-tuning capabilities as well.
These run on your local machine or can I use Colab/Huggingface GPUs?
This is the readme for the one I mentioned: https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Run...
> can I use Colab/Huggingface GPUs?
You use these templates on the runpod platform itself. Theres no free tier equivalent like you have with Colab/HF, but currently you can rent an RTX 4090 for $0.69/hr so its pretty affordable.
Do you know if that is possible?
[1] https://huggingface.co/docs/hub/spaces-sdks-docker-first-dem...
If 13B is good... I wonder if this will catch on in the finetuning community.
People care less about the LLaMA license than you'd think, and this is also about the time new models with "improved" architectures (like Falcon) should start popping up.
My knowledge of neural nets and AI is just lacking.
https://user-images.githubusercontent.com/48489457/243093269...
You can see that "Perplexity" goes down as Model Size goes up.
Their point was a lot of research that had been showing jumps in performance at certain "breakpoints" for lack of a better word were the result of badly selected metrics versus a case of suddenly emergent behaviour.
The nice thing about that research is it suggests that if you are able to try something on a smaller model it will scale nicely to a bigger model.
The key insight from the abstract, "Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance."
Yup that is another recent and interesting development for sure!
Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).
Their point was a lot of papers use those weird metrics and it contributes to the appearance of emergent ability, when in reality its just the bad metrics.
Nothing you've said so far disagrees with either my understanding or the conclusion of the paper I linked.
Several factors influence performance beyond parameter count, notable ones include: training corpus quality, training flops, and the downstream task.
It depends on how much compute you are spending on training and how big of a model you’re talking about.
There’s a “minimum” tokens/parameter for increasing size to be effective at improving loss/perplexity, so as you go up in parameters you generally will have to broaden your corpus which may lower it’s quality (e.g. tweets/reddit posts vs books/articles).
This effect isn’t as significant at 65B parameters as there is still enough high quality training data but if you’re talking 1T the corpus will (probably, I haven’t tried this/seen this done) by necessity be significantly poorer quality as you will overfit by simply repeating (it’s only beneficial so many times to repeat).
As a general rule, when validation loss/perplexity are the same in two models of different sizes downstream performance seems to be also generally the same (this was briefly explored in the PaLM 2 paper by Google) although it doesn’t correlate perfectly.
Practically speaking, this translates into a bigger model is better for applications we’re generally talking about. It just may not hold infinitely which we’re starting to see evidence of.
It’s definitely not linear though, you can look at some of the OpenLLaMA benchmarks (without getting into the weeds of if current benchmarks are representative) and the accuracy improvements in even 13B is not that significant (noting here all models were trained for the smear number of tokens, so relatively overtraining the smaller models).
There are probably some threshold parameter sizes that make a big difference but it’s still being determined.
https://ai.google/static/documents/palm2techreport.pdf
But note thst Meta's LLaMA 33B and 65B were trained with more tokens (1.2 trillion?) than the 13B and 7B models (1 trillion).
And subjectively, the larger parameter models do indeed feel "smarter" beyond what objective metrics would suggest.
Weights being subject to copyright was never tested in court. Businesses have wisely steered away from using leaked weights for commercial purposes. With OpenLLaMA we can expect novel products that incorporate these small but capable LLMs, for example as game NPCs.
https://huggingface.co/models?sort=downloads&search=13B
It is a mess, but its pretty much the only destination for finetunes from various communities/institutions.
This led to differences in evals, now they are bringing in more modern chips.
Aside from some tokeniser differences this is a drop in replacement for existing LLaMA that matches performance.
I don't think 30b and 65b are useful given what we do, the key is optimising models for consumer & swarming them.
As for SD.. Maybe try the bot on the discord server testing the new version: https://discord.com/invite/stablediffusion
The jump in quality is very significant as well.
65b is definitely a lot less common.
What other differences are there? The page doesn't mention how numbers are handled. These are two major things that impact model reasoning and numeric ability.
We can now set and forget runs so will have a better dataset for the next 13b and different tokeniser, this was meant to match as close as possible to be a drop in.
Native fine tuning is still out of consumer reach for the forseeable future, but there's people experimenting with QLORAs. The pipeline is still relatively new though and is a bit involved.
https://koboldai.org/discord
https://github.com/LostRuins/koboldcpp
I only have an older MacBook so I’m not sure what I can install.
The 20b model is 780b tokens in, lots of learnings so we can optimise future runs.
Hopefully these will be useful bases for continued research, we will have some SFT/RLHF variants in due course from our Carper AI lab.
Being fast at generating "something" (spitting out tokens) is one thing but... if those tokens aren't worth much (aka if the quality of the "answers" are weak), what is it really worth?
So, GPT4 is, well, way more than 100B+.
And yes, I share your opinion
Most of the valuable data in the world is private data unsuitable to send to GPT4 and other proprietary models.
How? Is there not extreme correlation between "13B parameters (low/not a lot) most likely means lots of hallucinations"
13B x10 might be the only feasible business model for your company, but that alone doesn't make it a superior option. Can you share more details on how the models work together?
I tried using some of these 13b models and was completely underwhelmed when compared gpt4.
Sam has said multiple times "the age of large models is over" https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
Either way, the LLaMa models are quite powerful and now that we have an open, commercially usable version in 3, 7, and 13b sizes is huge. The 65b model I'm assuming is coming soon as well.
For some, nothing. For some, a lot.
If you’re looking to create a ChatGPT replacement, this ain’t it.
If you’re trying to learn about LLMs, how they work, capabilities vs deficiencies, private data use, etc., these are invaluable to get your hands dirty in a private environment.
That may/will allow you to raise funds which will allow you to actually pay for OpenAI’s API or maybe train your own LLM or whatever.
Again, if you think in terms of MVP, these become extremely valuable.
Got any examples? I don't understand how it can "hallucinate like crazy" for some but not for all
2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334
How usable is it on MacBook Air / Pro, and how much GB RAM is required?
And regarding programming, how comparable is it to GPT-3.5 / GPT-4?
Loaded into 27.7GB of VRAM, requiring an A100 (without quantization).
Inferences are speedy, looks promising for a local solution compared to other models which have been released recently.
Is there any kind of standardized test to gauge the quality (not the speed) of LLM answers? aka, how hard does it hallucinate?
https://github.com/EleutherAI/lm-evaluation-harness
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
GGML 13b models at 4bit (Q4_0) take somewhere around 9gb of ram and q5_K_M take about 11gb. Gpu offloading support has also been added, I've been using 22 layers on my laptop rtx 2070 max q 8gb vram and CLBlast. I get around ~2-3 tokens per second with 13b models. In my experience, running 13b models is worth the extra time it takes to generate a response compared to 7b models. GPTQ is faster, I think, but I can't fit a quantized 13b model so I don't use it.
TheBloke [2] has been quantizing models and uploading them to HF and will probably upload a quantized version of this online soon. His discord server also has good guides to help you get going, linked in the model card of most of his models.
https://github.com/LostRuins/koboldcpp
https://huggingface.co/TheBloke
Edit: There's a bug with the newest Nvidia drivers that causes speed slowdown with large context size. I downgraded and stayed on 531.61 . The theory is that newer drivers change how out of cuda memory management works when trying to avoid OOM errors.
https://www.reddit.com/r/LocalLLaMA/comments/1461d1c/major_p...
https://github.com/vladmandic/automatic/discussions/1285
Moreover, you can fine-tune model for your specific tasks and you need fewer resources to fine tune a smaller model.
However, fine tuned small models can outperform general purpose large models on specific tasks.
There are also many lightweight tasks like basic sentiment analysis where the correctness of small models can be good enough to point of being indistinguishable from large models.
[1] https://arxiv.org/pdf/2302.13971.pdf
AFAIK the largest available models are for non-commercial use only.
This is a big limitation to the ´LLMs are entering their stable diffusion moment’ narrative.
What I'd really like to see improve is having larger contexts. The current context size available for llama.cpp-compatible models is 2048 tokens, which given the scenario and character descriptions at the start of the prompt only gives the LLM about 3 paragraphs of memory. It just forgets anything you said more than 3 paragraphs ago, which makes for a pretty miserable longer term RP experience unless you constantly update the summary of what's happened in the story/roleplay so far to be sent with every prompt.
4096 or larger contexts running efficiently (>= 1 T/s) with 12GB/24GB consumer grade GPU layer offloading would be fantastic, and bonus points if we can get 30 or 40B models working with that.