This extrapolates based on a good set of data points to predict when AI will reach significant milestones like being able to “work on tasks for a full 8 hours” (estimates by 2026). Which is ok - but it bears keeping https://xkcd.com/605/ in mind when doing extrapolation.
> Instead, even a relatively conservative extrapolation of these trends suggests that 2026 will be a pivotal year for the widespread integration of AI into the economy:
> Models will be able to autonomously work for full days (8 working hours) by mid-2026.
At least one model will match the performance of human experts across many industries before the end of 2026.
> By the end of 2027, models will frequently outperform experts on many tasks.
First commandment of tech hype: the pivotal, groundbreaking singularity is always just 1-2 years away.
I mean seriously, why is that? Even when people like OP try to be principled and use seemingly objective evaluation data, they find that the BIG big thing is 1-2 years away.
Self driving cars? 1-2 years away.
AR glasses replacing phones? 1-2 years away.
All of us living our life in the metaverse? 1-2 years away.
Again, I have to commend OP on putting in the work with the serious graphs, but there’s something more at play here.
Is it purely a matter of data cherry picking? Is it the unknowns unknowns leading to the data driven approaches being completely blind to their medium/long term limitations?
Exponential curves don't last for long fortunately, or the universe would have turned into a quark soup. The example of COVID is especially ironic, considering it stopped being a real concern within 3 years of its advent despite the exponential growth in the early years.
Those who understand exponentials should also try to understand stock and flow.
> Given consistent trends of exponential performance improvements over many years and across many industries, it would be extremely surprising if these improvements suddenly stopped.
I'm sure people were saying that about commercial airline speeds in the 1970's too.
But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.
With LLM's at the moment, the limiting factors might turn out to be training data, cost, or inherent limits of the transformer approach and the fact that LLM's fundamentally cannot learn outside of their context window. Or a combination of all of these.
The tricky thing about S curves is, you never know where you are on them until the slowdown actually happens. Are we still only in the beginning of the growth part? Or the middle where improvement is linear rather than exponential? And then the growth starts slowing...
>> it would be extremely surprising if these improvements suddenly stopped.
> But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.
An S-curve is exactly the opposite of "suddenly" stopping.
It is possible for us to get a sudden stop, due to limiting factors.
For a hypothetical: if Moore's Law had continued until we hit atomic resolution instead of the slowdown as we got close to it, that would have been an example of a sudden stop: can't get transistors smaller than atoms, but yet it would have been possible (with arbitrarily large investments that we didn't have) to halve transistor sizes every 18 months until suddenly we can't.
Now I think about it, the speed of commercial airlines is also an example of a sudden stop: we had to solve sonic booms first before even considering a Concorde replacement.
Ironically, given that it probably mistakes a sigmoid curve for an exponential curve, "Failing to understand the exponential, again" is an extremely apt name for this blog post.
> Again we can observe a similar trend, with the latest GPT-5 already astonishingly close to human performance:
Yes but only if you measure "performance" as "better than the other option more than 50% of the time" which is a terrible way to measure performance, especially for bullshitting AI.
Imagine comparing chocolate brands. One is tastier than the other one 60% of the time. Clear winner right? Yeah except it's also deadly poisonous 5% of the time. Still tastier on average though!
As they say, every exponential is a sigmoid in disguise. I think the exponential phase of growth for LLM architectures is drawing to a close, and fundamentally new architectures will be necessary for meaningful advances.
I'm also not convinced by the graphs in this article. OpenAI is notoriously deceptive with their graphs, and as Gary Marcus has already noted, that METR study comes with a lot of caveats: [https://garymarcus.substack.com/p/the-latest-ai-scaling-grap...]
A lot of this post relies on the recent open ai result they call GDPval (link below). They note some limitations (lack of iteration in the tasks and others) which are key complaints and possibly fundamental limitations of current models.
But more interesting is the 50% win rate stat that represents expert human performance in the paper.
That seems absurdly low, most employees don’t have a 50% success rate on self contained tasks that take ~1 day of work. That means at least one of a few things could be true:
1. The tasks aren’t defined in a way that makes real world sense
2. The tasks require iteration, which wasn’t tested, for real world success (as many tasks do)
I think while interesting and a very worthy research avenue, this paper is only the first in a still early area of understanding how AI will affect with the real world, and it’s hard to project well from this one paper.
> By the end of 2027, models will frequently outperform experts on many tasks.
In passing the quiz-es
> Models will be able to autonomously work for full days (8 working hours) by mid-2026.
Who will carry responsibility for the consequences of these model's errors? What tools will be avaiable to that resposible _person_?
--
Tehchno optimists will be optimistic. Techno pessimists will be pessimistic.
Processes we're discussing have their own limiting factors which no one mentiones. Why to mention what exactly makes graph go up and holds it from going exponential? Why to mention or discuss inherit limitations of the LLMs architecture? Or what is legal perspective on AI agency?
Thus we're discussing results of AI models passing tests and people's perception of other people opinions.
It's interesting that he brings up the example of "exponential" growth in the case of COVID infections even though it was actually logistic growth[1] that saturates once resources get exhausted. What makes AI different?
IMO this approach ultimately asks the wrong question. Every exponential trend in history has eventually flattened out. Every. single. one. Two rabbits would create a population with a mass greater than the Earth in a couple of years if that trend continues indefinitely. The left hand side of a sigmoid curve looks exactly like exponential growth to the naked eye... until it nears the inflection point at t=0. The two curves can't be distinguished when you only have noisy data from t<0.
A better question is, "When will the curve flatten out?" and that can only be addressed by looking outside the dataset for which constraints will eventually make growth impossible. For example, for Moore's law, we could examine as the quantum limits on how small a single transistor can be. You have to analyze the context, not just do the line fitting exercise.
The only really interesting question in the long term is if it will level off at a level near, below, or above human intelligence. It doesn't matter much if that takes five years or fifty. Simply looking at lines that are currently going up and extending them off the right side of the page doesn't really get us any closer to answering that. We have to look at the fundamental constraints of our understanding and algorithms, independent of hardware. For example, hallucinations may be unsolvable with the current approach and require a genuine paradigm shift to solve, and paradigm shifts don't show up on trend lines, more or less by definition.
I think the first comment on the article put it best: With COVID, researchers could be certain that exponential growth was taking place because they knew the underlying mechanisms of the growth. The virus was self-replicating, so the more people were already infected, the faster would new infections happen.
(Even this dynamic would only go on for a certain time and eventual slow down, forming an S-curve, when the virus could not find any more vulnerable persons to continue the rate of spread. The critical question was of course if this would happen because everyone was vaccinated or isolated enough to prevent infection - or because everyone was already infected or dead)
With AI, there is no such underlying mechanism. There is the dream of the "self-improving AI" where either humans can make use of the current-generation AI to develop the next-generation AI in a fraction of the time - or where the AI simply creates the next generation on its own.
If this dream were reality, it could be genuine exponential growth, but from all I know, it isn't. Coding agents speed up a number of bespoke programming tasks, but they do not exponentially speed up development of new AI models. Yes, we can now quickly generate large corpora of synthetic training data and use them for distillation. We couldn't do that before - but a large part of the training data discussion is about the observation that synthetic data can not replace real data, so data collection remains a bottleneck.
There is one point where a feedback loop does happen, and this is with the hype curve: Initial models produced extremely impressive results compared to everything we had before - there caused an enormous hype and unlocked investments that allowed more resources for the developed of the next model - which then delivered even better results. But it's obvious that this kind of feedback loop will eventually end when no more additional capital is available and diminishing returns set in.
Then we will once again be in the upper part of the S-curve.
"Models will be able to autonomously work for full days (8 working hours)" does not make them equivalent to a human employee. My employees go home and come back retaining context from the previous day; they get smarter every month. With Claude Code I have to reset the context between bite-sized tasks.
To replace humans in my workplace, LLMs need some equivalent of neuroplasticity. Maybe it's possible, but it would require some sort of shift in the approach that may or may not be coming.
Just because something exhibits an exponential growth at one point in time, that doesn’t mean that a particular subject is capable of sustaining exponential growth.
Their Covid example is a great counter argument to their point in that covid isn’t still growing exponentially.
Where the AI skeptics (or even just pragmatists, like myself) chime in is saying “yeah AI will improve. But LLMs are a limited technology that cannot fully bridge the gap between what they’re producing now, and what the “hypists” claim they’ll be able to do in the future.”
People like Sam Altman know ChatGPT is a million miles away from AGI. But their primary goal is to make money. So they have to convince VCs that their technology has a longer period of exponential growth than what it actually will have.
>they somehow jump to the conclusion that AI will never be able to do these tasks at human level
I don’t see that, I mostly see AI criticism that it’s not up to the hype, today. I think most people know it will approach human ability, we just don’t believe the hype that it will be here tomorrow.
I’ve lived through enough AI winter in the past to know that the problem is hard, progress is real and steady, but we could see a big contraction in AI spending in a few years if the bets don’t pay off well in the near term.
The money going into AI right now is huge, but it carries real risks because people want returns on that investment soon, not down the road eventually.
>People notice that while AI can now write programs, design websites, etc, it still often makes mistakes or goes in a wrong direction, and then they somehow jump to the conclusion that AI will never be able to do these tasks at human levels, or will only have a minor impact. When just a few years ago, having AI do these things was complete science fiction!
Both things can be true, since they're orthogonal.
Having AI do these things was complete fiction 10 years ago. And after 5 years of LLM AI, people do start to see serious limits and stunted growth with the current LLM approaches, while also seeing that nobody has proposed another serious contended to that approach.
Similarly, going to the moon was science finction 100 years ago. And yet, we're now not only not in Mars, but 50+ years without a new moon manned landing. Same for airplanes. Science fiction in 1900. Mostly stale innovation wise for the last 30 years.
A lot of curves can fit an exponential line plot, without the progress going forward being exponential.
We would have 1 trillion transistor cpus following Moore's "exponential curve"
Aside from the S-versus-exp issue, this area is one of these things where there's a kind of disconnect between my personal professional experience with LLMs and the criteria measures he's talking about. LLMs to me have this kind of superficially impressive feel where it seems impressive in its capabilities, but where, when it fails, it fails dramatically, in a way humans never would, and it never gets anywhere near what's necessary to actually be helpful on finishing tasks, beyond being some kind of gestalt template or prototype.
I feel as if there needs to be a lot more scrutiny on the types of evaluation tasks being provided — whether they are actually representative of real-world demands, or if they are making them easy to look good, and also more focus on the types of failures. Looking through some of the evaluation tasks he links to I'm more familiar with, they seem kind of basic? So not achieving parity with human performance is more significant than it seems. I also wonder, in some kind of maxmin sense, whether we need to start focusing more on worst-case failure performance rather than best-case goal performance.
LLMs are really amazing in some sense, and maybe this essay makes some points that are important to keep in mind as possibilities, but my general impression after reading it is it's kind of missing the core substance of AI bubble claims at the moment.
79 comments
[ 3.1 ms ] story [ 71.8 ms ] thread> Models will be able to autonomously work for full days (8 working hours) by mid-2026. At least one model will match the performance of human experts across many industries before the end of 2026.
> By the end of 2027, models will frequently outperform experts on many tasks.
First commandment of tech hype: the pivotal, groundbreaking singularity is always just 1-2 years away.
I mean seriously, why is that? Even when people like OP try to be principled and use seemingly objective evaluation data, they find that the BIG big thing is 1-2 years away.
Self driving cars? 1-2 years away.
AR glasses replacing phones? 1-2 years away.
All of us living our life in the metaverse? 1-2 years away.
Again, I have to commend OP on putting in the work with the serious graphs, but there’s something more at play here.
Is it purely a matter of data cherry picking? Is it the unknowns unknowns leading to the data driven approaches being completely blind to their medium/long term limitations?
I have issues with "human performance" as single data point in times where education keeps to excel in some countries and degrades in others.
How far away are we from saying, better than "X percent of humans" ?
Those who understand exponentials should also try to understand stock and flow.
I'm sure people were saying that about commercial airline speeds in the 1970's too.
But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.
With LLM's at the moment, the limiting factors might turn out to be training data, cost, or inherent limits of the transformer approach and the fact that LLM's fundamentally cannot learn outside of their context window. Or a combination of all of these.
The tricky thing about S curves is, you never know where you are on them until the slowdown actually happens. Are we still only in the beginning of the growth part? Or the middle where improvement is linear rather than exponential? And then the growth starts slowing...
Also elegantly formulated by: https://idlewords.com/talks/web_design_first_100_years.htm
> But a lot of technologies turn out to be S-shaped, not purely exponential, because there are limiting factors.
An S-curve is exactly the opposite of "suddenly" stopping.
It is possible for us to get a sudden stop, due to limiting factors.
For a hypothetical: if Moore's Law had continued until we hit atomic resolution instead of the slowdown as we got close to it, that would have been an example of a sudden stop: can't get transistors smaller than atoms, but yet it would have been possible (with arbitrarily large investments that we didn't have) to halve transistor sizes every 18 months until suddenly we can't.
Now I think about it, the speed of commercial airlines is also an example of a sudden stop: we had to solve sonic booms first before even considering a Concorde replacement.
Just because the model fits so far does not mean it will continue to fit.
Yes but only if you measure "performance" as "better than the other option more than 50% of the time" which is a terrible way to measure performance, especially for bullshitting AI.
Imagine comparing chocolate brands. One is tastier than the other one 60% of the time. Clear winner right? Yeah except it's also deadly poisonous 5% of the time. Still tastier on average though!
I'm also not convinced by the graphs in this article. OpenAI is notoriously deceptive with their graphs, and as Gary Marcus has already noted, that METR study comes with a lot of caveats: [https://garymarcus.substack.com/p/the-latest-ai-scaling-grap...]
Like the improvements between 60 and 70 felt much faster than those between 80 and 90.
But more interesting is the 50% win rate stat that represents expert human performance in the paper.
That seems absurdly low, most employees don’t have a 50% success rate on self contained tasks that take ~1 day of work. That means at least one of a few things could be true:
1. The tasks aren’t defined in a way that makes real world sense
2. The tasks require iteration, which wasn’t tested, for real world success (as many tasks do)
I think while interesting and a very worthy research avenue, this paper is only the first in a still early area of understanding how AI will affect with the real world, and it’s hard to project well from this one paper.
https://cdn.openai.com/pdf/d5eb7428-c4e9-4a33-bd86-86dd4bcf1...
In passing the quiz-es
> Models will be able to autonomously work for full days (8 working hours) by mid-2026.
Who will carry responsibility for the consequences of these model's errors? What tools will be avaiable to that resposible _person_?
--
Tehchno optimists will be optimistic. Techno pessimists will be pessimistic.
Processes we're discussing have their own limiting factors which no one mentiones. Why to mention what exactly makes graph go up and holds it from going exponential? Why to mention or discuss inherit limitations of the LLMs architecture? Or what is legal perspective on AI agency?
Thus we're discussing results of AI models passing tests and people's perception of other people opinions.
[1] https://en.wikipedia.org/wiki/Logistic_function#Modeling_ear...
https://situational-awareness.ai/from-gpt-4-to-agi/
IMO this approach ultimately asks the wrong question. Every exponential trend in history has eventually flattened out. Every. single. one. Two rabbits would create a population with a mass greater than the Earth in a couple of years if that trend continues indefinitely. The left hand side of a sigmoid curve looks exactly like exponential growth to the naked eye... until it nears the inflection point at t=0. The two curves can't be distinguished when you only have noisy data from t<0.
A better question is, "When will the curve flatten out?" and that can only be addressed by looking outside the dataset for which constraints will eventually make growth impossible. For example, for Moore's law, we could examine as the quantum limits on how small a single transistor can be. You have to analyze the context, not just do the line fitting exercise.
The only really interesting question in the long term is if it will level off at a level near, below, or above human intelligence. It doesn't matter much if that takes five years or fifty. Simply looking at lines that are currently going up and extending them off the right side of the page doesn't really get us any closer to answering that. We have to look at the fundamental constraints of our understanding and algorithms, independent of hardware. For example, hallucinations may be unsolvable with the current approach and require a genuine paradigm shift to solve, and paradigm shifts don't show up on trend lines, more or less by definition.
I think the first comment on the article put it best: With COVID, researchers could be certain that exponential growth was taking place because they knew the underlying mechanisms of the growth. The virus was self-replicating, so the more people were already infected, the faster would new infections happen.
(Even this dynamic would only go on for a certain time and eventual slow down, forming an S-curve, when the virus could not find any more vulnerable persons to continue the rate of spread. The critical question was of course if this would happen because everyone was vaccinated or isolated enough to prevent infection - or because everyone was already infected or dead)
With AI, there is no such underlying mechanism. There is the dream of the "self-improving AI" where either humans can make use of the current-generation AI to develop the next-generation AI in a fraction of the time - or where the AI simply creates the next generation on its own.
If this dream were reality, it could be genuine exponential growth, but from all I know, it isn't. Coding agents speed up a number of bespoke programming tasks, but they do not exponentially speed up development of new AI models. Yes, we can now quickly generate large corpora of synthetic training data and use them for distillation. We couldn't do that before - but a large part of the training data discussion is about the observation that synthetic data can not replace real data, so data collection remains a bottleneck.
There is one point where a feedback loop does happen, and this is with the hype curve: Initial models produced extremely impressive results compared to everything we had before - there caused an enormous hype and unlocked investments that allowed more resources for the developed of the next model - which then delivered even better results. But it's obvious that this kind of feedback loop will eventually end when no more additional capital is available and diminishing returns set in.
Then we will once again be in the upper part of the S-curve.
To replace humans in my workplace, LLMs need some equivalent of neuroplasticity. Maybe it's possible, but it would require some sort of shift in the approach that may or may not be coming.
Their Covid example is a great counter argument to their point in that covid isn’t still growing exponentially.
Where the AI skeptics (or even just pragmatists, like myself) chime in is saying “yeah AI will improve. But LLMs are a limited technology that cannot fully bridge the gap between what they’re producing now, and what the “hypists” claim they’ll be able to do in the future.”
People like Sam Altman know ChatGPT is a million miles away from AGI. But their primary goal is to make money. So they have to convince VCs that their technology has a longer period of exponential growth than what it actually will have.
I don’t see that, I mostly see AI criticism that it’s not up to the hype, today. I think most people know it will approach human ability, we just don’t believe the hype that it will be here tomorrow.
I’ve lived through enough AI winter in the past to know that the problem is hard, progress is real and steady, but we could see a big contraction in AI spending in a few years if the bets don’t pay off well in the near term.
The money going into AI right now is huge, but it carries real risks because people want returns on that investment soon, not down the road eventually.
If AI was so great, why all curl hackerone submissions have been rejected? Slop is not a substitute of skill.
AI agents' performance depends heavily on the context / data / environment provided, and how that fits into the overall business process.
Thus, "agent performance" itself will be very unevenly distributed.
Both things can be true, since they're orthogonal.
Having AI do these things was complete fiction 10 years ago. And after 5 years of LLM AI, people do start to see serious limits and stunted growth with the current LLM approaches, while also seeing that nobody has proposed another serious contended to that approach.
Similarly, going to the moon was science finction 100 years ago. And yet, we're now not only not in Mars, but 50+ years without a new moon manned landing. Same for airplanes. Science fiction in 1900. Mostly stale innovation wise for the last 30 years.
A lot of curves can fit an exponential line plot, without the progress going forward being exponential.
We would have 1 trillion transistor cpus following Moore's "exponential curve"
Cerebras wafer scale chip has 4 trillion transistors.
https://www.cerebras.ai/chip
I feel as if there needs to be a lot more scrutiny on the types of evaluation tasks being provided — whether they are actually representative of real-world demands, or if they are making them easy to look good, and also more focus on the types of failures. Looking through some of the evaluation tasks he links to I'm more familiar with, they seem kind of basic? So not achieving parity with human performance is more significant than it seems. I also wonder, in some kind of maxmin sense, whether we need to start focusing more on worst-case failure performance rather than best-case goal performance.
LLMs are really amazing in some sense, and maybe this essay makes some points that are important to keep in mind as possibilities, but my general impression after reading it is it's kind of missing the core substance of AI bubble claims at the moment.