> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
Like any algorithm, it is dependent on the quality of its training data. In a study, a group of academics from New York University found 40% of the code produced by Copilot had cybersecurity flaws.
Yeah let me use this tool where 40% of its output is garbage. Toss this in the ever-growing AI Winter bin.
Given the fact that Copilot is trained on a corpus of real code, is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is? If human-written code has about the same 40% suffering from security issues then then using Copilot is certainly no worse. It definitely doesn't seem fair to invalidate the idea of AI-assisted code generation without testing that.
> is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is?
It sounds to me like that's not an inference that can easily be drawn. Copilot was trained on predicting code, it doesn't understand the code it produces syntactically. Security issues can be highly context dependent. For example, in most cases it's fine to log a variable, but when it happens to contain a password, it's a security issue. This is a flawed example as the algorithm may be able to learn that variables with names or contexts suggesting that they're secrets should not be logged, but I can imagine much more subtle issues can crop up.
That doesn't seem to be the claim. They say one third was written with AI help.
Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it.
edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.
I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.
I mean, Google search uses AI, and developers use Google to find things, so technically you could say the vast majority of code is written with "AI help", but that's not exactly a noteworthy observation to make...
I work for GitHub (although writing in an entirely personal capacity) and the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help.
I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4
I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.
So 30% of code written by users with the extension installed was written with some extension assistance?
How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed?
My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.
> the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help.
The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism.
I need to write an article titled "People who install Copilot only use it for 30% of the code".
Maybe they're just measuring lines committed (total guess)?
If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there.
Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.
>Nearly a third of new code on GitHub is written with AI help
May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.
Martin from GitHub here. While Copilot is good it's not quite up to those numbers yet I'm afraid. It’s early days, but we’re already seeing that for some languages when people have it installed, we see about 30% of newly written code being suggested by Copilot. Vary’s between languages and the numbers of users in the technical preview is a LOT less than the 73m folks we have on GitHub, so a long way to go before it makes up a third of everything..
This number just doesn't feel remotely plausible. Hasn't Copilot supported just one editor and five languages until now? And it's advertised as being available to a limited number of testers, and gated behind a waitlist. Oh, and according to the article half the people who were granted access to Copilot have stopped using it.
There is no way to make these numbers work unless we assume that the people using Copilot are committing orders of magnitude more code. And if they are, how likely is it that they're really legit users?
> 40% of the code produced by Copilot had cybersecurity flaws.
Off-topic, but it's quite ironic to me that "cyber" had such a resurgence. In ~2010 to 2015, most people in my circle considered cyber to be a washed out joke, and yet, in the last few years, it gained massive popularity again. It's still hard for me to take anything "cyer-" seriously, though.
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
reply
(Emphasis mine)
I wonder if what this really means is ~30% of code written in codespaces or the web/cloud version of vscode you can use within a repo is written with co-pilot. Which I find much easier to believe than 30% of all code uploaded to github was written with AI assistance (how would they ever be able to measure that).
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
Maybe, all in all, we spend 33% of our time writing very dumb code that can easily written by some AI ? (I must recognize that a good chunk of the code I was writing as a professipnal dev was boiler plate stuff...)
Most of AI tools like TabNine, Copilot or Kite are quite resource intensive.
For example I have TabNine, but it is active ONLY when my laptop is plugged in, otherwise it drains batteries pretty quickly.
That's why I don't believe that more than 30% of developers are OK to heat up their laptops for getting AI suggestions. It more looks like a marketing title
I find it weird that people keep slamming copilot. IDEA (and other IDEs) have for years auto complete for for loops, functions and a few other things. The most this tool does is put in a couple of extra like 'user.someInterestingProperty' based on your context inside those same for loop shells. It's just that the for-loop shell is now, for better or worse, similar to what people write.
I've seen some of the suggestions in my workflow at this point and it's much slower than IDE assist stuff, and generally barely helpful. So I don't see any reason to slam the crap out of it.
I wonder if it's a mix of people feeling like they A) will lose their job, B) make the software around the world worse (it's already horrendously bad) or C) hate Microsoft?
I can't really tell why anyone would honestly hate this tool other than a lot of pent up whateverism.
I've always been somewhat suspicious of this AI generated code stuff. I get what's happening, but I'm not convinced it's all that fundamentally different than what compilers have been doing for decades.
Isn't a compiler a code generator? And isn't the stuff we humans write simply instructions to a compiler on what code to generate?
Compilers have tons of intelligence in them. At least when I was a CS grad (an ancient time ago) complex sets of heuristics were considered AI. And compilers use complex sets of heuristics to determine things like optimizations.
So I guess what I'm saying is that this news feels over hyped to me.
50 comments
[ 3.6 ms ] story [ 132 ms ] threadOr maybe they just count what share of commiters have the Coopilot extension installed ...
Yeah let me use this tool where 40% of its output is garbage. Toss this in the ever-growing AI Winter bin.
It sounds to me like that's not an inference that can easily be drawn. Copilot was trained on predicting code, it doesn't understand the code it produces syntactically. Security issues can be highly context dependent. For example, in most cases it's fine to log a variable, but when it happens to contain a password, it's a security issue. This is a flawed example as the algorithm may be able to learn that variables with names or contexts suggesting that they're secrets should not be logged, but I can imagine much more subtle issues can crop up.
Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it.
edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
The fact that the combo reached 30% of newly generated code doesn't make sense.
https://www.youtube.com/watch?v=EfYMKOMc738
I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4
I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.
How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed?
My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.
The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism.
I need to write an article titled "People who install Copilot only use it for 30% of the code".
That is a far different headline than the one on this post. WTH is wrong with journalists now days? FFS.
If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there.
Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.
>Nearly a third of new code on GitHub is written with AI help
May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.
Insofar as 0%-1% is a value compatible with “as much as 30%” e.g. <30%
There is no way to make these numbers work unless we assume that the people using Copilot are committing orders of magnitude more code. And if they are, how likely is it that they're really legit users?
Off-topic, but it's quite ironic to me that "cyber" had such a resurgence. In ~2010 to 2015, most people in my circle considered cyber to be a washed out joke, and yet, in the last few years, it gained massive popularity again. It's still hard for me to take anything "cyer-" seriously, though.
(Emphasis mine)
I wonder if what this really means is ~30% of code written in codespaces or the web/cloud version of vscode you can use within a repo is written with co-pilot. Which I find much easier to believe than 30% of all code uploaded to github was written with AI assistance (how would they ever be able to measure that).
where is the reference link?
a third of all code committed by active copilot users has been accepted copilot suggestions…
I've seen some of the suggestions in my workflow at this point and it's much slower than IDE assist stuff, and generally barely helpful. So I don't see any reason to slam the crap out of it.
I wonder if it's a mix of people feeling like they A) will lose their job, B) make the software around the world worse (it's already horrendously bad) or C) hate Microsoft?
I can't really tell why anyone would honestly hate this tool other than a lot of pent up whateverism.
> For some languages, we already see about 30% of newly written code being suggested by Copilot.
It writes about 30% of Python and 20-25% for other languages.
Isn't a compiler a code generator? And isn't the stuff we humans write simply instructions to a compiler on what code to generate?
Compilers have tons of intelligence in them. At least when I was a CS grad (an ancient time ago) complex sets of heuristics were considered AI. And compilers use complex sets of heuristics to determine things like optimizations.
So I guess what I'm saying is that this news feels over hyped to me.