A lot of companies were betting billions on exponential growth, or at least hoping that 2024 wouldn't be when AI performance starts to converge. Language models are still glorified chatbot that can't be taken seriously, and are more of a liability than anything useful. Remember Air Canada being on the hook for the incorrect information its chatbot gave? They tried to contest it, lost, and removed the bot from their site.
The post seems to be about changing the Leaderboard and doesn't comment too much about whether the actual real-life performance of LLMs is plateauing and what can be done about it.
From what I got from this, they have picked a bunch of benchmarks where many LLMs can easily get to about 90%. It does not seem to be possible for an LLM to exceed about 95% on any sort of benchmark given the randomness in the process. The lack of progress on these easy problems means that HuggingFace's leaderboard is not as meaningful.
They are sort of saying both things: LLMs are plateauing, but the benchmarks are also too easy.
Even if they’re plateauing there’s still a lot of value to be had in what they already do. I think the mistake so far has been aiming too high or too low—ie, products that require AGI-like LLMs or unimaginative “low-hanging fruit” product ideas which are obvious from the function of an LLM. The former have been wishful thinking, and the latter have no moat. The Goldilocks area is in understanding what current LLMs can do in a way that you can either do something complicated that we couldn’t do reliably without LLMs or do something simple that wasn’t worth doing without LLMs. And in both cases the products need to be built in a way that naturally incorporates the expected failure modes of the tech. (For example, I don’t need it to write all my code; there’s a lot of value in just using ChatGPT to help me write one-off bash scripts.)
Just to be sure: the post does not say that the performance of Open LLMs is plateauing (because that would be false, e.g. Google just released a Gemma2 that blows out of the water all previous open models from the same sizes).
Its true title is "Performances are plateauing, let's make the leaderboard steep again", which means "on the Open LLM leaderboard, top models have basically reached a point where they've all grokked the benchmarks which makes it harder to distinguish them, so let's change the benchmarks for harder ones to make a difference again."
11 comments
[ 3.7 ms ] story [ 39.1 ms ] threadThe post seems to be about changing the Leaderboard and doesn't comment too much about whether the actual real-life performance of LLMs is plateauing and what can be done about it.
They are sort of saying both things: LLMs are plateauing, but the benchmarks are also too easy.
https://open-llm-leaderboard-blog.static.hf.space/dist/index...
The only thing that actually lives on the page at the submission URL is the little floating breadcrumb at the top right.
Its true title is "Performances are plateauing, let's make the leaderboard steep again", which means "on the Open LLM leaderboard, top models have basically reached a point where they've all grokked the benchmarks which makes it harder to distinguish them, so let's change the benchmarks for harder ones to make a difference again."