> Of course, it’s unclear how exactly Microsoft and Google are measuring what’s AI-generated versus not, so these figures are best taken with a grain of salt.
This does seem to me to be the key question, is anyone transparent about this? If not, why not?
Only if it's effective at writing code, which it of course is not.
Any time you see companies refusing to even vaguely define what metrics like this mean (or, for that matter, using non-standard metrics, like disclosing weekly active users but not monthly), it's generally a very strong signal that they're not interested in being transparent because the truth is, ah, not what they would like it to be.
The perception of having fallen behind in AI adversely impacts your stock price, the amount of capital you can marshal to actually compete in AI. What I think is actually happening industry-wide is that any sort of "intelligence" in software is slowly being rebranded as AI.
it also drives my gmail sentence autocomplete. it does not mean “30% of my email is written by AI”. It does help me type faster though. Reframing the one as the other is, imo, securities fraud. (I will asterisk that YC startups vibecoding their product is real, but that’s, like, 10^10 lower LOC scale than “all of Google”)
IMO, the key thing that software engineers want to know with these numbers is "is there still an engineer involved." In my mind LLM powered autocomplete that generates a lot of code is just totally different from "PM says they want this feature and the AI generates the entire thing from scratch" in that one amplifies the capabilities of an engineer while the other replaces them.
really really really depends on what sort of code is being 'written'. 20 years ago, IDEs would automatically create boilerplate getters and setters. In large projects that's a non-trivial amount of code. IDEs can autocomplete stuff already. For most of the folks I know doing non-trivial projects, AI tools are... useful autocompletes, but not much more. So... 25% of your code was done by AI but is it the hard nitty gritty stuff? The value prop of your whole company? Or is it just lots of boilerplate that is necessary because of all the abstractions we have at our disposal today (or... all the abstractions that are required to do anything 'modern' to use a negative light on it)?
How many lines in a diff are actually relevant code? Anyone who does reviews knows the answer.
That is one of the reasons why lean, terse languages are often better to review.
We can guess by those companies preferred coding styles and technologies whether their codebases are lean and terse or full of straw. And that should give us an estimate.
Of course, I could be wrong. They could be doing this measurement after removing irrelevant changes.
The better choice would be not to publish those sorts of claims if there is not a clear methodology that explains how the number was achieved.
It doesn’t have to be contiguous AI-written chunks. This could also mean X% accepted AI suggestions on Y% codebase where X*Y < 30%.
AI suggestions can also be as simple as autocomplete and still be counted for the sake of engagement metrics.
Oh and in enterprise settings and especially MS shops Github Copilot is being pushed everywhere, (forceful) adoption rates are much higher than the market average.
If any line of any codebase was written by AI, then up to 30% can be true, in the way that my ISP will claim ~900 Mbps still qualifies as up to 2 Gbps.
> Satya Nadella said that 20%-30% of code inside the company’s repositories was “written by software”
and
> The Microsoft CEO said the company was seeing mixed results in AI-generated code across different languages, with more progress in Python and less in C++.
So the CEO of Microsoft is saying that 20 - 30% of their code is being produced by computer systems that write poor code?
Horrible why would you not let people eat on earth and in the name of saving you take away jobs. Let the AI because helpful as a tool to help people not to just take away jobs.
> "I'd say maybe 20%, 30% of the code that is inside of our repos today and some of our projects are probably all written by software," Nadella said during a conversation before a live audience with Meta CEO Mark Zuckerberg.
which is clearly untrue, one assumes he meant "20%, 30% written since 2023 was partially generated by an LLM operated by a developer", but that doesn't sell stock.
Generated by software -> generated by AI seems huge logical leap. Then again maybe it can be for given meaning of "AI".
Not that 30% of code being automatically generated from templates or in some algorithmic way seem unbelievable. There is likely lot of code that could be generated by other code and it might even be reasonable choice.
40 comments
[ 4.7 ms ] story [ 108 ms ] threadThis does seem to me to be the key question, is anyone transparent about this? If not, why not?
Any time you see companies refusing to even vaguely define what metrics like this mean (or, for that matter, using non-standard metrics, like disclosing weekly active users but not monthly), it's generally a very strong signal that they're not interested in being transparent because the truth is, ah, not what they would like it to be.
How many lines in a diff are actually relevant code? Anyone who does reviews knows the answer.
That is one of the reasons why lean, terse languages are often better to review.
We can guess by those companies preferred coding styles and technologies whether their codebases are lean and terse or full of straw. And that should give us an estimate.
Of course, I could be wrong. They could be doing this measurement after removing irrelevant changes.
The better choice would be not to publish those sorts of claims if there is not a clear methodology that explains how the number was achieved.
AI suggestions can also be as simple as autocomplete and still be counted for the sake of engagement metrics.
Oh and in enterprise settings and especially MS shops Github Copilot is being pushed everywhere, (forceful) adoption rates are much higher than the market average.
[0] Claude 3.7 in my recent experience
and
> The Microsoft CEO said the company was seeing mixed results in AI-generated code across different languages, with more progress in Python and less in C++.
So the CEO of Microsoft is saying that 20 - 30% of their code is being produced by computer systems that write poor code?
Does it mean AI though? Lots of lines in repositories are generated by software that isn’t AI. Dependency lock files, proto files, etc
IMO the wording is intentionally misleading.
> "I'd say maybe 20%, 30% of the code that is inside of our repos today and some of our projects are probably all written by software," Nadella said during a conversation before a live audience with Meta CEO Mark Zuckerberg.
which is clearly untrue, one assumes he meant "20%, 30% written since 2023 was partially generated by an LLM operated by a developer", but that doesn't sell stock.
There is no use of writing VBA these days :@
Not that 30% of code being automatically generated from templates or in some algorithmic way seem unbelievable. There is likely lot of code that could be generated by other code and it might even be reasonable choice.