I watched the Google interview with Ilya yesterday and this came up. There's a large disconnect between the evals and the real-world performance, and he admitted that the evals are targeted.
There was a storm of hype the last couple weeks for Gemini 3 and everyone, correctly, rolled their eyes. Investors are demanding a return and it's not happening. They're just going to have to face reality at some point.
Unfortunately, investors facing reality means homelessness for the rest of us. Maybe the real treasure was the billions of dollars we made along the way. :)
What is the definition of "soaring"? The charts in the article showed that the percentage of the companies that adopt AI for automation has increase 3X. At least 40% of the companies pay for GenAI, and at least 10% of the employees use GenAI daily. Combined with the fact that the companies like OpenAI and Anthronpic frequently run out of capacity, how is the AI use not soaring?
As soon as every big corp started stuffing their UIs with AI buttons, we all knew it was investors pushing for AI use to go sky high without a care for the nuances of the current state of AI. The reality is that AI usage isn't as impactful as it was promised. Where is the productivity increase in being able to generate a picture via some prompt? When deep research could contain hallucinated text or references, where is the productivity increase? It is undeniable that these tools have uses but when you look at all the investment made into this tech, the outcomes are not great.
Is it not soaring? I can't think of a recent time a new technology was invented and I began using it almost every day, and I don't even consider myself that heavy of a user of AI.
> A survey by Dayforce, a software firm, finds that while 87% of executives use ai on the job, just 57% of managers and 27% of employees do. Perhaps middle managers set up ai initiatives to satisfy their superiors’ demands, only to wind them down quietly at a later date.
The article quietly ignored two better explanations: the day to day work of executives can be automated more easily (Manna vibes) and/or the execs have a vested interest in AI succeeding so they can cut headcount so they are evangelists for AI.
Investors don't "expect" AI to soar, they NEED AI to soar. Why are we still engaging in this absolutely ridiculous kabuki theatre? This entire cracking edifice is propped up by fictitious capital and pipe dreams, and the music is about to stop. Turns out, you can't charge $20 a month for something no one wants and expect to get a trillion dollars out of it. Shocker!
I am using it for ansible, php, java, c, linux configuration issues or general questions. Preparing excel sheets etc..
It's sped the time I need to produce projects from a usual span of 4-20 days to 1-2 days with another 2-3 Testing. Of course I still bill the time it would have taken me but for a professional it can be a great improvement.
While my country will be slow to adopt, we haven't even adopted to smartphones yet - hooray Germany, it will have to adopt eventually ( in 10 years or so )
> Three years into the generative-AI wave, demand for the technology looks surprisingly flimsy.
Lets compare to the adoption of the internet. Mosaic was released in 1993. Businesses adopted the internet progressively during the 90s, starting slow but accelerating toward the decade's end with broad adoption of the internet as a business necessity by 2000.
Three years is a ridiculously small amount of time for businesses to make dramatic changes.
Honestly, I sometimes use heavy thinking models only to avoid wasting tokens on my expensive pro plans. In many cases, I prefer using quicker models to discuss something, gain better ideas about the topic, do "my own research" using Google, then discuss again. ChatGPT Pro mode is helpful, but it's too slow, and I have no idea what they say is true (even with sources) or I'm familiar with the topic so I can research faster myself.
I use coding agents often, but I don't burn all the tokens out of my Claude Max plan and ChatGPT Business plan with two seats.
Maybe the archive link stripped it out, but it would be really useful to look at the actual sources, because TFA seems to be, uhh, "selective" in what stats it presents. For example, this source (Alex Bick at the St Louis Federal Bank) seems to be cited:
TFA presents the most pessimistic stat it could find: daily GenAI usage at work growing from 12.1% to 12.6% in a year. (Interestingly there was a dip to 9% in Nov 2024; maybe end-of-year holidays?)
It does not mention that the same tracker also shows that overall usage (at and outside work, at least once last week) has steadily climbed from 44% to 54%. That is a 10 percentage point growth in a year. (This may also be why OpenAI reveals WAU rather than DAU; people mostly regularly use it on a weekly basis.)
Here is something even more interesting from the same authors at the St Louis Fed using the same data:
Really, read that article, it is short and a bit astounding. Money quote:
> When we feed these estimates into a standard aggregate production model, this suggests that generative AI may have increased labor productivity by up to 1.3% since the introduction of ChatGPT. This is consistent with recent estimates of aggregate labor productivity in the U.S. nonfarm business sector. For example, productivity increased at an average rate of 1.43% per year from 2015-2019, before the COVID-19 pandemic. By contrast, from the fourth quarter of 2022 through the second quarter of 2025, aggregate labor productivity increased by 2.16% on an annualized basis. Relative to its prepandemic trend, this corresponds to excess cumulative productivity growth of 1.89 percentage points since ChatGPT was publicly released.
...
...
> We stress that this correlation cannot be interpreted as causal, and that labor productivity is determined by many factors. However, the current results are suggestive that generative AI may already be noticeably affecting industry-level productivity.
A lot of people who think AI is being used heavily, are coders. It's like a blacksmith making a hammer for himself and thinking that everyone is using the hammer everyday all the time.
Let's check agentic AI. Which agents do people mostly talk about? Aha - coding agents!
I like how the framing of the article assumes that AI is a revolutionary technology that everyone should be using and the adoption is just mysteriously slow. This was particularly funny:
> In recent earnings calls, nearly two-thirds of executives at S&P 500 companies mentioned AI. At the same time, the people actually responsible for implementing AI may not be as forward-thinking, perhaps because they are worried about the tech putting them out of a job.
Ah, those brave, forward-looking executives with their finger on the pulse of the future while their employees are just needlessly stalling adoption. Completely absent from the article is the possibility that the technology is not as revolutionary as claimed.
All very interesting. Lean startup anyone? As a "serial" startup founder one thing that was always hammered into my brain was: customer discovery! Clearly something that was drowned out by all the money thrown at AI startups.Motto: we just buy our customers.
all new technology is overestimated in the short term, and underestimated in the long term.
(soar - overestimated)
for example, years ago in the era of dragon naturallyspeaking, ALL computers would momentarily be using speech recognition.
and it didn't happen
but quietly speech recognition started working in the background - call trees on the phone, and other places where a strict vocabulary could help. it quietly grew and nowadays it is everywhere.
29 comments
[ 3.0 ms ] story [ 37.2 ms ] threadThere's no way that could work. $x expands to multiple arguments.
There was a storm of hype the last couple weeks for Gemini 3 and everyone, correctly, rolled their eyes. Investors are demanding a return and it's not happening. They're just going to have to face reality at some point.
did he do something other than that podcast?
The article quietly ignored two better explanations: the day to day work of executives can be automated more easily (Manna vibes) and/or the execs have a vested interest in AI succeeding so they can cut headcount so they are evangelists for AI.
It's sped the time I need to produce projects from a usual span of 4-20 days to 1-2 days with another 2-3 Testing. Of course I still bill the time it would have taken me but for a professional it can be a great improvement.
While my country will be slow to adopt, we haven't even adopted to smartphones yet - hooray Germany, it will have to adopt eventually ( in 10 years or so )
This is the point.
This is what matters.
A revolutionary technology birthed in a bonfire of cash
Lets compare to the adoption of the internet. Mosaic was released in 1993. Businesses adopted the internet progressively during the 90s, starting slow but accelerating toward the decade's end with broad adoption of the internet as a business necessity by 2000.
Three years is a ridiculously small amount of time for businesses to make dramatic changes.
People are captivated by good stories, and AI makes for one hell of a sci fi narrative
It's hard to separate the maybe one day plausible fictional future from the on-the-ground reality
I use coding agents often, but I don't burn all the tokens out of my Claude Max plan and ChatGPT Business plan with two seats.
https://www.genaiadoptiontracker.com/
TFA presents the most pessimistic stat it could find: daily GenAI usage at work growing from 12.1% to 12.6% in a year. (Interestingly there was a dip to 9% in Nov 2024; maybe end-of-year holidays?)
It does not mention that the same tracker also shows that overall usage (at and outside work, at least once last week) has steadily climbed from 44% to 54%. That is a 10 percentage point growth in a year. (This may also be why OpenAI reveals WAU rather than DAU; people mostly regularly use it on a weekly basis.)
Here is something even more interesting from the same authors at the St Louis Fed using the same data:
https://www.stlouisfed.org/on-the-economy/2025/nov/state-gen...
Really, read that article, it is short and a bit astounding. Money quote:
> When we feed these estimates into a standard aggregate production model, this suggests that generative AI may have increased labor productivity by up to 1.3% since the introduction of ChatGPT. This is consistent with recent estimates of aggregate labor productivity in the U.S. nonfarm business sector. For example, productivity increased at an average rate of 1.43% per year from 2015-2019, before the COVID-19 pandemic. By contrast, from the fourth quarter of 2022 through the second quarter of 2025, aggregate labor productivity increased by 2.16% on an annualized basis. Relative to its prepandemic trend, this corresponds to excess cumulative productivity growth of 1.89 percentage points since ChatGPT was publicly released. ... ...
> We stress that this correlation cannot be interpreted as causal, and that labor productivity is determined by many factors. However, the current results are suggestive that generative AI may already be noticeably affecting industry-level productivity.
Let's check agentic AI. Which agents do people mostly talk about? Aha - coding agents!
> In recent earnings calls, nearly two-thirds of executives at S&P 500 companies mentioned AI. At the same time, the people actually responsible for implementing AI may not be as forward-thinking, perhaps because they are worried about the tech putting them out of a job.
Ah, those brave, forward-looking executives with their finger on the pulse of the future while their employees are just needlessly stalling adoption. Completely absent from the article is the possibility that the technology is not as revolutionary as claimed.
(soar - overestimated)
for example, years ago in the era of dragon naturallyspeaking, ALL computers would momentarily be using speech recognition.
and it didn't happen
but quietly speech recognition started working in the background - call trees on the phone, and other places where a strict vocabulary could help. it quietly grew and nowadays it is everywhere.