The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer.
My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc.
FWIW I saw a similar number for myself, around 30% more PRs in the last 6 months, compared to the 6 months before that (I picked up agentic coding around at the start of the year). And a similar increase for closed issues.
In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support. This means they were low priority. However, many were of the cleanup/refactor type, so they might result in speedups later.
Yeah welcome to the state of the art in measuring AI impact. I have contacts at a few different larger tech companies that are fully AI pilled (the one I work at included) and every single one has forgotten the last 50 years of lessons in measuring dev productivity and hyperfocused on PR throughput and token usage.
Fun fact: all the data I've seen suggests at most a 50% uplift in those metrics. And that's at the top percentiles. Its very clear that the already high performers see the greatest uplift but anyone in that meaty middle will only see incremental gains.
My personal experience was that I saw AI adopters opening and merging more PRs but I also spent more of my day reviewing PRs than I did in the past, in order to keep up with the flood of new stuff. Reconcile that with MSFT's repeated layoffs however you like.
To me the biggest gain I see is that you take the programmers out of the loop. Instead of formulating your ideas to start a project and then acquiring the resources to do a single iteration on it which may take months if not years, now many specialists "just do it" and do several such iterations in a single day. Afterwards they may still go the ordinary route via programmers but on a completely different level and a lot of fruitless work is frontloaded and 100x cheaper. This doesn't show up in any such statistic.
So, the question is not "Does it make our programmers more productive?" but "Does it make our organization faster?".
more features doesn't mean anything if they don't translate into economic value, refactoring prs and dependency updates are easily automated now and it saves alot of productivity, but if you compare the costs and economic value gained it might not actually justify enterprise token spending.
>281 billion tokens. Using the least expensive version of Claude Opus 4.6, which costs $5 for every million tokens, that one
user alone could have cost Meta more than $1.4 million.
And that is based on the pricing that they sell a dollar for less than a dollar so the costs may be much much more.
I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are also 24% boosted or else it's just an accounting trick.
The number of merged PRs increased. But what does that actually tell us?
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.
Context: measuring productivity at all is hard. It was hard when computers were introduced into the workplace, when the internet was introduced, etc.
I worked with a journalist who covered tech in the 80s & 90s and she remembered that studies then didn't show improvements in productivity with the adoption of computers, despite people feeling as though they were more productive.
Reminds me of this Jeff Bezos quote:
"The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There's something wrong with the way you are measuring it."
12 comments
[ 33.3 ms ] story [ 852 ms ] threadMy guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc.
FWIW I saw a similar number for myself, around 30% more PRs in the last 6 months, compared to the 6 months before that (I picked up agentic coding around at the start of the year). And a similar increase for closed issues.
In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support. This means they were low priority. However, many were of the cleanup/refactor type, so they might result in speedups later.
Fun fact: all the data I've seen suggests at most a 50% uplift in those metrics. And that's at the top percentiles. Its very clear that the already high performers see the greatest uplift but anyone in that meaty middle will only see incremental gains.
So, the question is not "Does it make our programmers more productive?" but "Does it make our organization faster?".
And that is based on the pricing that they sell a dollar for less than a dollar so the costs may be much much more.
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.
I worked with a journalist who covered tech in the 80s & 90s and she remembered that studies then didn't show improvements in productivity with the adoption of computers, despite people feeling as though they were more productive.
Reminds me of this Jeff Bezos quote:
"The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There's something wrong with the way you are measuring it."