LLMs make me 10-20x more productive in frontend work which I barely do.
But when it comes to low-level stuff (C/C++) I personally don't find it too useful. it just replaces my need to search stackoverflow.
edit: should have mentioned the low-level stuff I work on is mature code and a lot of times novel.
> To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study.
So it's a small sample size of 16 developers. And it sounds like different tasks were (randomly) assigned to the no-AI and with-AI groups - so the control group doesn't have the same tasks as the experimental group. I think this could lead to some pretty noisy data.
Interestingly - small sample size isn't in the list of objections that the auther includes under "Addressing Every Objection You Thought Of, And Some You Didn’t".
I do think it's an interesting study. But would want to see if the results could be reproduced before reading into it too much.
As others probably have experienced, I can only add that I am doing coding now I would have kicked down the road if I did not have LLM assistance.
Example: using LeafletJS — not hard, but I didn't want to have to search all over to figure out how to use it.
Example: other web page development requiring dropping image files, complicated scrolling, split-views, etc.
In short, there are projects I have put off in the past but eagerly begin now that LLMs are there to guide me. It's difficult to compare times and productivity in cases like that.
What bothers me more than any of this particular discussion is that we seem to be incapable of determining programmer productivity in a meaningful way since my debut as a programmer 40 years ago.
I’ve been around tech for a long time. At this point, I’ve lost count of how many hype cycles I’ve seen hit the “hold on, everything sucks” stage. Generative AI is seemingly at the hold on, everything sucks stage and it’s getting repetitive.
What if this is true? And then we as a developer community are focused on the wrong thing to increase productivity?
Like what if by focusing on LLMs for productivity we just reinforce old-bad habits, and get into a local maxima... And even worse, what if being stuck with current so-so patterns, languages, etc means we don't innovate in language design, tooling, or other areas that might actually be productivity wins?
They averaged producing 47% more code on the AI tasks, but took only 20% more time. The report here biases over these considerations, but I’m left wondering: was the extra code superfluous or did this produce better structure / managed debt better? If that extra 47% of code translates to lower debt and more consistent throughput over the long term, I might take it, given how crushed projects get from debt. Anyway, it’s all hyperbole because there are massive statistical differences in the outcomes but no measures as to what they mean, but I’m sure they have meaning. That meaning matters a ton.
Now do a study that specifically gauges how useful an LLM (including smart tab completion) is for a frontend dev working in react/next/tailwind on everyday Jira tickets.
These were maintainers of large open source projects. It's all relative. It's clearly providing massive gains for some and not as much for others. It should follow that it's benefit to you depends on who you are and what you are working on.
I think this for me is the most worrying: "You can see that for AI Allowed tasks, developers spent less time researching and writing code".
My analogy to this is seeing people spend time trying to figure out how to change colors, draw shapes in powerpoint, rather than focus on the content and presentation. So here, we have developers now focusing their efforts on correcting the AI output, rather than doing the research and improving their ability to deliver code in the future.
This entire concept hinges on AI not getting better. If you believe AI is going continue to get better at the current ~5-10% a month range, then hand waiving over developer productivity today is about the same thing as writing an article about the internet being a fad in 1999.
I find LLMs are decent at regurgitating boilerplate. Basically the same kind of stuff you could google then copy-paste... AI chatbots, now that they have web access, are also good at going over documentation and save you a little time searching through the docs yourself.
They're not great at business logic though, especially if you're doing anything remotely novel. Which is the difficult part of programming anyway.
But yeah, to the average corporate programmer who needs to recreate the same internal business tool that every other company has anyway, it probably saves a lot of time.
I finally took the plunge and did a big chunk of work in Cursor. It was pretty ideal: greenfield but with a very relevant example to slightly modify (the example pulled events over HTTP as a server and I wanted it to pull events over Google pub/sub instead).
Over IDK, 2-3 hours I got something that seemed on its face to work, but:
- it didn't use the pub/sub API correctly
- the 1 low-coverage test it generated didn't even compile (Go)
- there were a bunch of small errors it got confused by--particularly around closures
I got it to "90%" (again though it didn't at all work) with the first prompt, and then over something like a dozen more mostly got it to fix its own errors. But:
- I didn't know the pub/sub API--I was relying on Cursor to do this correctly--and it totally submarined me
- I had to do all the digging to get the test to compile
- I had to go line by line and tell it to rewrite... almost everything
I quit when I realized I was spending more time prompting it to fix things than it would take me to fully engage my brain and fix them myself. I also noticed that there was a strong pull to "just do one more prompt" rather than dig in and actually understand things. That's super problematic to me.
Worse, this wasn't actually faster. How do I know that? The next day I did what I normally do: read docs and wrote it myself. I spent less time (I'm a fast typist and a Vim user) overall, and my code works. My experience matches pretty well w/ the results of TFA.
---
Something I will say though is there is a lot of garbage stuff in tech. Like, I don't want to learn Terraform (again) just to figure out how to deploy things to production w/o paying a Heroku-like premium. Maybe I don't want to look up recursive CTEs again, or C function pointers, or spent 2 weeks researching a heisenbug I put into code for some silly reason AI would have caught immediately. I am _confident_ we can solve these things without boiling oceans to get AI to do it for us.
But all this shit about how "I'm 20x more productive" is totally absurd. The only evidence we have of this is people just saying it. I don't think a 20x productivity increase is even imaginable. Overall productivity since 1950 is up 3.6x [0]. These people are asking us to believe they've achieved over 400 years of productivity gains in "3 months". Extraordinary claims require extraordinary evidence. My guess is either you were extremely unproductive before, or (like others are saying in the threads) in very small ways you're 20x more productive but most things are unaffected or even slower.
The more I used it, the easier it became to skip over things I should have thought through myself. But looking back, the results weren’t always faster or better.
Now I prefer to treat AI as a kind of challenger. It helps reveal the parts I haven't truly understood, rather than just speeding things up.
AI could make me more productive, I know that for a fact. But, I don't want to be more productive because the tasks that could be automated with AI are those I find enjoyable. Not always in an intellectual sense, but in a meditative sense. And if I automated those away, I think I would become less human.
I think the dichotomy you see with how positive people are about ai has almost entirely to do with the kind of questions they ask.
That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useless.
Conversely there's people who love ai and use it for everything, and since the majority of the stuff they ask about is fairly simple and well documented (eg "Write me some typescript"), they rarely have a negative experience.
I think you touched on an important aspect, but did not explore it further.
If we accept that AI is a tool, then then problem is the nature of the tool as it will vary heavily from individual to individual. This partially accounts for the ridiculous differences from self reported accounts of people, who use it on a regular basis.
And then, there is a possibility that my questions are not that unusual and/or are well documented ( quite possible ) so my perception of the usefulness of those answers is skewed.
My recent interaction with o4 was pretty decent on a very new ( by industry standards ) development and while documentation for it exists, it is a swirling vortex of insanity from where I sit. I was actually amazed to see how easily 4o saw some of those discrepancies and listed those to me along with likely pitfalls that may come with it. We will be able to find if that prediction holds v.soon.
> You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useless.
You have any example questions where o3 failed to be helpful?
I use it pretty similarly to you, only resorting to it to unblock myself basically, otherwise I'm mostly the one doing the actual work, LLMs help with specific functions or specific blockers, or exploring new "spaces". But almost all the times I've gotten stuck, o3 (and o3-pro mode) managed to unstuck me, once I've figured out the right way to ask the question, even when my own searching and reading didn't help.
I had to create a Cython module wrapping some C, used Claude 4 and GPT 4.1, they were worse than useless. One can imagine why I needed help with that project.
It's kind of true. I only use it for simple stuff that I don't have time for. For example, how to write a simple diagram in tikz. The Ai does the simple and busywork of providing a good enough approximation which I can tweak and get what I want.
For hard questions, I prefer to use my own skills, because AI often regurgitates what I'm already aware. I still ask AI in the off-chance it comes up with something cool, but most often, I have to do it myself.
Well, I use it before google, since it in general summarizes webpages and removes the ads. Quite handy.
It’s also very useful to check if you understand something correctly. And for programming specifically I found it really useful to help naming stuff (which tends to be hard not in the least place because it’s subjective).
Perhaps is difficult to measure personal productivity in programming, but we can measure that we will run more slowly with 10 kg. in our backpack. I propose this procedure: The SWE selects 10 tasks and guesses some measure of their complexity (time to finish them) and then he randomly select 5 to be done with AI and the rest without. He performs them and finally calculates a deviation D. The deviation D = D_0 - D_1 where D_i = sum (real_time/guessed_time - 1), where D_0 is using AI and D_1 is without AI, the sign and magnitude of D measure respectively if the use of AI is beneficial or detrimental and the impact of using AI. Also, clipping individuals addends to be in the interval [-0.5,0.5] should avoid one bad guess dominating the estimation. Sorry if this is a trivial ideal but it is feasible and intuitively should provide useful information if the tasks are taken among the ones in which each initial guessing has small deviation. A filter should be applied to tasks in which scaffolding by AI surpass a certain relative threshold in case we are interested in generalizing our results to tasks in which scaffolding is not dominating time.
It could happen that the impact of using AI depends of the task at hand, the capability of the SWE to pair programming with it, and of the LLM used, to such an extend that those factors were bigger that the average effect on a bag of tasks, in this case the large deviation from the mean makes any one parameter estimation void of useful information.
I have never found a measure of programmer productivity that makes sense to me, but I can say that LLM coding tools are way more distracting to me than they are worth. They constantly guess at what I may type next, are often wrong, and pop in with suggestions breaking my mental flow and making me switch from the mindset of coding to the mindset of reviewing code.
The article brushed aside devs being terrible at estimates, but I dunno.
I'm a frontend guy, been using Claude Code for a couple of weeks now. It's been able to speed up some boilerplate, it's sped up a lot of "naming is hard" conversations I like to have (but my coworkers probably don't, lol), it's enabled me to do a lot more stuff in my most recent project.
But for a task or two I suspect that it has slowed me down. If I'm unable to articulate the problem well enough and the problem is hard enough you can go in circles for awhile. And I think the nature of "the right answer is just around the corner" makes it hard to timebox or find a specific point where you say "yup, time to ditch this and do it the old-fashioned way". There is a bit of a slot-machine effect here.
> To compute the actual speedup – or, rather, slowdown! – provided by AI tools, the researchers compared the developers’ predictions of how long each task would take to the measured completion time.
I'm sorry, but it feels to me like this research has only proven that developers tend to underestimate how long a task is supposed to take, with or without AI.
In no way did they actually measure how much faster a specific task was when performed with and without AI?
37 comments
[ 4.0 ms ] story [ 61.6 ms ] threadedit: should have mentioned the low-level stuff I work on is mature code and a lot of times novel.
> To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study.
So it's a small sample size of 16 developers. And it sounds like different tasks were (randomly) assigned to the no-AI and with-AI groups - so the control group doesn't have the same tasks as the experimental group. I think this could lead to some pretty noisy data.
Interestingly - small sample size isn't in the list of objections that the auther includes under "Addressing Every Objection You Thought Of, And Some You Didn’t".
I do think it's an interesting study. But would want to see if the results could be reproduced before reading into it too much.
I thought it was the model, but then I realised, v0 is carried by the shadcn UI library, not the intelligence of the model
Example: using LeafletJS — not hard, but I didn't want to have to search all over to figure out how to use it.
Example: other web page development requiring dropping image files, complicated scrolling, split-views, etc.
In short, there are projects I have put off in the past but eagerly begin now that LLMs are there to guide me. It's difficult to compare times and productivity in cases like that.
Like what if by focusing on LLMs for productivity we just reinforce old-bad habits, and get into a local maxima... And even worse, what if being stuck with current so-so patterns, languages, etc means we don't innovate in language design, tooling, or other areas that might actually be productivity wins?
These were maintainers of large open source projects. It's all relative. It's clearly providing massive gains for some and not as much for others. It should follow that it's benefit to you depends on who you are and what you are working on.
It isn't black and white.
My analogy to this is seeing people spend time trying to figure out how to change colors, draw shapes in powerpoint, rather than focus on the content and presentation. So here, we have developers now focusing their efforts on correcting the AI output, rather than doing the research and improving their ability to deliver code in the future.
Hmm...
They're not great at business logic though, especially if you're doing anything remotely novel. Which is the difficult part of programming anyway.
But yeah, to the average corporate programmer who needs to recreate the same internal business tool that every other company has anyway, it probably saves a lot of time.
How I measure performance is how many features I can implement in a given period of time.
It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.
Over IDK, 2-3 hours I got something that seemed on its face to work, but:
- it didn't use the pub/sub API correctly
- the 1 low-coverage test it generated didn't even compile (Go)
- there were a bunch of small errors it got confused by--particularly around closures
I got it to "90%" (again though it didn't at all work) with the first prompt, and then over something like a dozen more mostly got it to fix its own errors. But:
- I didn't know the pub/sub API--I was relying on Cursor to do this correctly--and it totally submarined me
- I had to do all the digging to get the test to compile
- I had to go line by line and tell it to rewrite... almost everything
I quit when I realized I was spending more time prompting it to fix things than it would take me to fully engage my brain and fix them myself. I also noticed that there was a strong pull to "just do one more prompt" rather than dig in and actually understand things. That's super problematic to me.
Worse, this wasn't actually faster. How do I know that? The next day I did what I normally do: read docs and wrote it myself. I spent less time (I'm a fast typist and a Vim user) overall, and my code works. My experience matches pretty well w/ the results of TFA.
---
Something I will say though is there is a lot of garbage stuff in tech. Like, I don't want to learn Terraform (again) just to figure out how to deploy things to production w/o paying a Heroku-like premium. Maybe I don't want to look up recursive CTEs again, or C function pointers, or spent 2 weeks researching a heisenbug I put into code for some silly reason AI would have caught immediately. I am _confident_ we can solve these things without boiling oceans to get AI to do it for us.
But all this shit about how "I'm 20x more productive" is totally absurd. The only evidence we have of this is people just saying it. I don't think a 20x productivity increase is even imaginable. Overall productivity since 1950 is up 3.6x [0]. These people are asking us to believe they've achieved over 400 years of productivity gains in "3 months". Extraordinary claims require extraordinary evidence. My guess is either you were extremely unproductive before, or (like others are saying in the threads) in very small ways you're 20x more productive but most things are unaffected or even slower.
[0]: https://fred.stlouisfed.org/series/OPHNFB
That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useless.
Conversely there's people who love ai and use it for everything, and since the majority of the stuff they ask about is fairly simple and well documented (eg "Write me some typescript"), they rarely have a negative experience.
If we accept that AI is a tool, then then problem is the nature of the tool as it will vary heavily from individual to individual. This partially accounts for the ridiculous differences from self reported accounts of people, who use it on a regular basis.
And then, there is a possibility that my questions are not that unusual and/or are well documented ( quite possible ) so my perception of the usefulness of those answers is skewed.
My recent interaction with o4 was pretty decent on a very new ( by industry standards ) development and while documentation for it exists, it is a swirling vortex of insanity from where I sit. I was actually amazed to see how easily 4o saw some of those discrepancies and listed those to me along with likely pitfalls that may come with it. We will be able to find if that prediction holds v.soon.
What I am saying is that it has its uses.
You have any example questions where o3 failed to be helpful?
I use it pretty similarly to you, only resorting to it to unblock myself basically, otherwise I'm mostly the one doing the actual work, LLMs help with specific functions or specific blockers, or exploring new "spaces". But almost all the times I've gotten stuck, o3 (and o3-pro mode) managed to unstuck me, once I've figured out the right way to ask the question, even when my own searching and reading didn't help.
For hard questions, I prefer to use my own skills, because AI often regurgitates what I'm already aware. I still ask AI in the off-chance it comes up with something cool, but most often, I have to do it myself.
It could happen that the impact of using AI depends of the task at hand, the capability of the SWE to pair programming with it, and of the LLM used, to such an extend that those factors were bigger that the average effect on a bag of tasks, in this case the large deviation from the mean makes any one parameter estimation void of useful information.
I'm a frontend guy, been using Claude Code for a couple of weeks now. It's been able to speed up some boilerplate, it's sped up a lot of "naming is hard" conversations I like to have (but my coworkers probably don't, lol), it's enabled me to do a lot more stuff in my most recent project.
But for a task or two I suspect that it has slowed me down. If I'm unable to articulate the problem well enough and the problem is hard enough you can go in circles for awhile. And I think the nature of "the right answer is just around the corner" makes it hard to timebox or find a specific point where you say "yup, time to ditch this and do it the old-fashioned way". There is a bit of a slot-machine effect here.
I'm sorry, but it feels to me like this research has only proven that developers tend to underestimate how long a task is supposed to take, with or without AI.
In no way did they actually measure how much faster a specific task was when performed with and without AI?