I'm happy to let people think that AI does not yield productivity gains. There is no point engaging on this topic, so I will just outwork/outperform them.
I think AI greatly reduces the starting costs for mundane tasks/boilerplate, for a reduction in velocity in implementation. So possibly an illusion that programmers feel more productive. It could be that RPE(Rate of perceived exertion) is lower when using AI for tasks, but raw throughput may be higher if programmers just do the jobs themselves and get into productive/flow state.
I'm building things at a level of complexity I wouldn't have even attempted without AI.
This piece, however, only focuses on time spent on a task that could be done both ways. Even there, it falls short. Let's assume this study is correct and a specific coding task does take me 19% more time with AI. I can still be more productive because the AI doing some of the work allows me to do other tasks during that time.
I do worry about atrophy of my mind outsourcing too many tasks, admittedly. But that's a different issue.
All AI impact studies and research papers need to be taken with a pinch of salt. The field is moving so fast that by the time you get peer reviewed, you’re already outdated
I’ve watched coding change from Cursor-esque IDEs to terminal based agentic tools within months.
It's not a bad paper, but it's also turning into a fantastic illustration of how much thirst there is out there for anything that shows that AI productivity doesn't work.
I find it funny that the first sentence is: I’m not sure if the whole AI thing is falling apart, but we are definitely in a phase where the hype doesn’t match what we are seeing in the real world.
Also I do not think AI is falling apart and I do not think the productivity loss is surprising. It's something I've noticed a lot, you have to check AI output line by line thereby making you slower. AI makes tasks easier, not necessarily faster. And because people are lazy and love making things easier it's still going to be adopted no matter what anyone says.
I had hardly written any code prior to ChatGPT other than teaching myself some VBA.
Since then, using Gen AI tools to learn + write code, I have deployed a functioning full stack application to the web. NextJS on Vercel, with a backend server also deployed running on Python, and a Supabase DB. Is it the best application ever making loads of money? Definitely not. Are there things wrong with it? Absolutely (although I promise I'm not exposing sensitive env vars and API keys to the web). Did the first versions look like absolute ass as I clumsily figured things out and made bad mistakes? You bet. But it's a functioning app that does some useful things and has real users.
I would never have imagined doing this in a million years prior to Gen AI.
Do some devs see mixed results depending on how they're using the tools? I'm sure. Is Gen AI overhyped broadly speaking? Probably so. But when I see people say it's a delusion, waste of resources, and everyone is wasting their time on it ... for me, it just doesn't line up.
Honestly, at this point, I'll just work 2 hours a day since I'm already producing 5-10x the output of my coworkers who aren't leveraging Claude Code at the medium-sized startup where I work. It really makes you wonder what's holding people back if they haven't at least doubled their productivity by now
I don't think AI lives up to the current hype but this article is garbage.
They're obviously talking about the METR paper, but the main takeaway according to the authors themselves was that self-reporting productivity increases is unreliable, not that you should cancel your subscription.
Nothing in that paper said that AI can't speed up software engineering.
The overarching problem is that signal:noise ratio in contemporary discussions of AI is absurdly low. This is thanks in large part to a media complex whose primary goal seems to be the pumping of AI investments and stocks, and the associated cacophony of mendacious exaggerations. If only we could have a reasonable discussion without the "contributions" of the non-technical and/or salespeople...
> “The results surprised us,” research lab METR reported. “Developers thought they were 20pc faster with AI tools, but they were actually 19pc slower when they had access to AI than when they didn’t.”
They forgot about training and practice. Try it with airplanes. "The results surprised us. Given Boeing 777 they thought they will get from London to New York in 7 hours, but they actually couldn't even get off the ground. In fact they couldn't even start the engines."
I think the productivity gains greatly depend on the coding task at hand. You could reasonably design a study that shows AI gains and slowdown depending on the task
There is really no argument that AI creates some productivity gains. Even if it's just an improved autocomplete. Because autocomplete does create some productivity gains. Pushing farther is murkier though. When it comes to the bulk of the type of work that AI is proving useful for, one of the main questions is why is there a need for speed. It's not in a sense of fear of automating jobs, it's just that generally we are reaching the bottlenecks more quickly and potentially causing more, but different problems, than we are solving.
It's similar to the story of the development of vehicles and how even though we move much faster we spend a greater amount of time in transit. My mom used to lament how annoying it was to have to drive to the grocery store because when she was younger and not everyone had cars the store came to you. Twice a day, in the morning and the evening the "rolling store" would drive through the neighborhood and if they didn't have what you needed right then, they would bring it on the next trip. We are finally coming back full circle with things like Instacart but it's taken a solid ~60 years of largely wasted inefficient travel times.
AI is NOT going away. Spend serious time to learn how leveraging AI can multiply your output ( quality and quantity ) or get left behind. Practice. Improve. Deliver. Period.
You’ve seen the posts and articles: “I’m inherently better than AI, coding faster and smarter.” Sure, maybe for now—especially if you’re half-assing your AI game. But that edge? It’s vanishing fast.
Stop being so self indulgent.
Become proficient with the tools available to you or get left behind.
> Just because you can deal with your inbox more quickly doesn’t mean you’ll spend your afternoon on the beach. The more emails you fire off, the more you’ll receive back, and the never-ending cycle continues.
I've seen this firsthand. It's so easy to produce "content" now, like emails, presentations, etc., that more of my time now is just sifting through AI slop looking for signal.
I'm on a board for a nonprofit, and we've seen a person that is upset with the organization send dozens of AI-generated quasi-legal demands full of hallucinations of rules and laws that don't actually exist. So now we're forced to pay a real human lawyer to deal with a denial of service attack of ChatGPT slop and are spending lots of time just dealing with the onslaught. Our productivity has crashed as a board due to a single bad actor enabled by ChatGPT.
At work, I'm also getting buried in AI sales calls and AI resumes. It's getting more and more difficult to find signal in the noise.
It wouldn't surprise me if there's a lot of people who are experiencing lower productivity for similar reasons.
24 comments
[ 2.3 ms ] story [ 41.1 ms ] threadThis piece, however, only focuses on time spent on a task that could be done both ways. Even there, it falls short. Let's assume this study is correct and a specific coding task does take me 19% more time with AI. I can still be more productive because the AI doing some of the work allows me to do other tasks during that time.
I do worry about atrophy of my mind outsourcing too many tasks, admittedly. But that's a different issue.
I’ve watched coding change from Cursor-esque IDEs to terminal based agentic tools within months.
It's not a bad paper, but it's also turning into a fantastic illustration of how much thirst there is out there for anything that shows that AI productivity doesn't work.
I just learned there was a 4 minute TV news segment about it on CNBC! https://www.youtube.com/watch?v=WP4Ird7jZoA
Also I do not think AI is falling apart and I do not think the productivity loss is surprising. It's something I've noticed a lot, you have to check AI output line by line thereby making you slower. AI makes tasks easier, not necessarily faster. And because people are lazy and love making things easier it's still going to be adopted no matter what anyone says.
Since then, using Gen AI tools to learn + write code, I have deployed a functioning full stack application to the web. NextJS on Vercel, with a backend server also deployed running on Python, and a Supabase DB. Is it the best application ever making loads of money? Definitely not. Are there things wrong with it? Absolutely (although I promise I'm not exposing sensitive env vars and API keys to the web). Did the first versions look like absolute ass as I clumsily figured things out and made bad mistakes? You bet. But it's a functioning app that does some useful things and has real users.
I would never have imagined doing this in a million years prior to Gen AI.
Do some devs see mixed results depending on how they're using the tools? I'm sure. Is Gen AI overhyped broadly speaking? Probably so. But when I see people say it's a delusion, waste of resources, and everyone is wasting their time on it ... for me, it just doesn't line up.
They're obviously talking about the METR paper, but the main takeaway according to the authors themselves was that self-reporting productivity increases is unreliable, not that you should cancel your subscription.
Nothing in that paper said that AI can't speed up software engineering.
Why are we responding to hype with nonsense?
They forgot about training and practice. Try it with airplanes. "The results surprised us. Given Boeing 777 they thought they will get from London to New York in 7 hours, but they actually couldn't even get off the ground. In fact they couldn't even start the engines."
What an utterly bizarre question. Yes, by definition being more productive means doing more work.
It's similar to the story of the development of vehicles and how even though we move much faster we spend a greater amount of time in transit. My mom used to lament how annoying it was to have to drive to the grocery store because when she was younger and not everyone had cars the store came to you. Twice a day, in the morning and the evening the "rolling store" would drive through the neighborhood and if they didn't have what you needed right then, they would bring it on the next trip. We are finally coming back full circle with things like Instacart but it's taken a solid ~60 years of largely wasted inefficient travel times.
You’ve seen the posts and articles: “I’m inherently better than AI, coding faster and smarter.” Sure, maybe for now—especially if you’re half-assing your AI game. But that edge? It’s vanishing fast.
Stop being so self indulgent.
Become proficient with the tools available to you or get left behind.
I've seen this firsthand. It's so easy to produce "content" now, like emails, presentations, etc., that more of my time now is just sifting through AI slop looking for signal.
I'm on a board for a nonprofit, and we've seen a person that is upset with the organization send dozens of AI-generated quasi-legal demands full of hallucinations of rules and laws that don't actually exist. So now we're forced to pay a real human lawyer to deal with a denial of service attack of ChatGPT slop and are spending lots of time just dealing with the onslaught. Our productivity has crashed as a board due to a single bad actor enabled by ChatGPT.
At work, I'm also getting buried in AI sales calls and AI resumes. It's getting more and more difficult to find signal in the noise.
It wouldn't surprise me if there's a lot of people who are experiencing lower productivity for similar reasons.
10% would mean a 10 trillion in productivity, 1% is 1 trillion.
https://sourcegraph.com/blog/the-brute-squad