What do they consider to be an "LLM of this size"?
While this technique of scaling up an existing pre-trained model via fine-tuning is really impressive, it feels a bit unfair to compare what's essentially now an 8.3B model to mostly standard 7B ones, especially considering how important scale is in predicting LLM performance.
I've just avoided engaging with the LLM "race" and stick with free ChatGPT 3.5 when I want to tinker. The technology is not on my critical path right now, so I'll probably engage when it settles into more defined products.
> I've just avoided engaging with the LLM "race" and stick with free ChatGPT 3.5 when I want to tinker.
In my experience, Phind also seems fairly nice and is free for now, at least when the Cloudflare protection in front of it doesn't fuss over me using a VPN: https://www.phind.com
It's nice that there are options that are basically "plug and play", both free and paid ones, if you don't feel like tinkering with the open source models, or don't have the hardware to run them with good performance (such as me only having an RX 580 GPU).
One of my biggest pet peeves with the LLM Benchmark culture is that in practice, marginal increases don't matter and are overall subjective. But using a strong model (like ChatGPT is still) is a good starting point.
The alternative approach is to start with a small[er] model, but derive reliable uncertainty estimates, only moving to a larger model if necessary (i.e., if the probability of the predictions is lower than needed for the task).
And I agree that the leaderboards don't currently reflect the quantities of interest typically needed in practice.
That is very, very hard to do in an objective manner, as the current LLM benchmark gaming demonstrates.
Sure, you can deploy a smaller model to production to get real-world user data and feedback, but a) deploying a suboptimal model can give a bad first impression and b) the quality is still subjective and requires other metrics to be analyzed. Looking at prediction probabilities only really helps if you have a single correct output token, which isn't what LLM benchmarks test for.
I believe we have two rather different settings in mind. My statement assumes the enterprise use-case, where having a verifier is required. (In this context, I'm also assuming the approach of constraining against the observed data.) In such a selective classification setting, the end-user need not be exposed to lower quality outputs, but rather null predictions if the model cascade has been exhausted (i.e., progressively moving to larger models until the probability is acceptable).
Hopefully in 2024 we can get at least one of the benchmarks to move to assessing non-parametric/distribution-free uncertainty for selective classification, reflecting more recent CS/Stats advances that should be used in practice. Working on it.
Same here. Official Mixtral & Mistral (the new v0.2) seem to have the best & most dependable output vs basically all their predecessors/derivatives of equal or smaller parameter sizes. Mixtral is on a whole other level with its baked-in 'chain of thought' reasoning.
The fine tunes (e.g. dolphin-mixtral, dolphin2.2-mistral, etc) may be good at coding or whatever else they specialize in, but they seem to sacrifice in other areas that result in 'hallucinating' where Mixtral/Mistral wouldn't - perhaps as a result of 'score chasing' on benchmarks like has been mentioned already.
Depends on your own specific needs & use cases whether this is important or not.
they started off building chatbot apps for teenagers. hugging face is this emoji -> https://emojipedia.org/hugging-face apparently teens are into this stuff.
then they pivoted into the whole ai thing like many other companies and are burning money now. the usual burn money till you get bought or you close the company with the owners getting rich of secondaries and your "lessons".
> then they pivoted into the whole ai thing like many other companies
I have nothing to do with the company, nor do I really use their platform, but I felt a need to point something out: your statement is a pretty egregious trivialisation of huggingface and their contributions to AI. They were releasing NLP papers in early 2018, and in the same year open-sourced one of the most impactful ML libraries in the industry's history. The original chatbot app was dropped very early on and their full focus became the research, tooling, and platform.
Hugging Face first gained popularity as they provided the first PyTorch-friendly implementations of BERT and GPT-2, which later became the transformers library.
They didn't pivot to AI, they were there since the start.
it's a horrific name but a decent company, it deeply annoys me to have to preface the name of a big player with a disclaimer about how stupid the name it when talking to people
“Developed by Tencent's ARC Lab, LLaMA-Pro is an 8.3 billion parameter model. It's an expansion of LLaMA2-7B, further trained on code and math corpora totaling 80 billion tokens.”
33 comments
[ 2.8 ms ] story [ 92.6 ms ] threadThere is also an -Instruct variant of this model published on HF.
They are claiming the best balance of performance on language and coding tasks in an open source LLM of this size.
Especially with keeping the original model weights the same.
While this technique of scaling up an existing pre-trained model via fine-tuning is really impressive, it feels a bit unfair to compare what's essentially now an 8.3B model to mostly standard 7B ones, especially considering how important scale is in predicting LLM performance.
In my experience, Phind also seems fairly nice and is free for now, at least when the Cloudflare protection in front of it doesn't fuss over me using a VPN: https://www.phind.com
Aside from that, I just paid for GitHub Copilot (https://github.com/features/copilot), albeit JetBrains are also coming out with their own product (https://www.jetbrains.com/ai/).
It's nice that there are options that are basically "plug and play", both free and paid ones, if you don't feel like tinkering with the open source models, or don't have the hardware to run them with good performance (such as me only having an RX 580 GPU).
One of my biggest pet peeves with the LLM Benchmark culture is that in practice, marginal increases don't matter and are overall subjective. But using a strong model (like ChatGPT is still) is a good starting point.
And I agree that the leaderboards don't currently reflect the quantities of interest typically needed in practice.
That is very, very hard to do in an objective manner, as the current LLM benchmark gaming demonstrates.
Sure, you can deploy a smaller model to production to get real-world user data and feedback, but a) deploying a suboptimal model can give a bad first impression and b) the quality is still subjective and requires other metrics to be analyzed. Looking at prediction probabilities only really helps if you have a single correct output token, which isn't what LLM benchmarks test for.
Hopefully in 2024 we can get at least one of the benchmarks to move to assessing non-parametric/distribution-free uncertainty for selective classification, reflecting more recent CS/Stats advances that should be used in practice. Working on it.
The fine tunes (e.g. dolphin-mixtral, dolphin2.2-mistral, etc) may be good at coding or whatever else they specialize in, but they seem to sacrifice in other areas that result in 'hallucinating' where Mixtral/Mistral wouldn't - perhaps as a result of 'score chasing' on benchmarks like has been mentioned already.
Depends on your own specific needs & use cases whether this is important or not.
then they pivoted into the whole ai thing like many other companies and are burning money now. the usual burn money till you get bought or you close the company with the owners getting rich of secondaries and your "lessons".
edit: here is an old article https://techcrunch.com/2017/03/09/hugging-face-wants-to-beco...
I have nothing to do with the company, nor do I really use their platform, but I felt a need to point something out: your statement is a pretty egregious trivialisation of huggingface and their contributions to AI. They were releasing NLP papers in early 2018, and in the same year open-sourced one of the most impactful ML libraries in the industry's history. The original chatbot app was dropped very early on and their full focus became the research, tooling, and platform.
Hugging Face first gained popularity as they provided the first PyTorch-friendly implementations of BERT and GPT-2, which later became the transformers library.
They didn't pivot to AI, they were there since the start.
“Developed by Tencent's ARC Lab, LLaMA-Pro is an 8.3 billion parameter model. It's an expansion of LLaMA2-7B, further trained on code and math corpora totaling 80 billion tokens.”