How much time are you spending trying to understand what it generated and checking whether it gave you something that performs well, or only performs well "right now but will grind to a half with even moderately more input"?
Most of what it generates for me I don't have to spend more than a second or two investigating. It seems to KNOW what type of db query I want to do next, it's quite uncanny.
There are times that it doesn't get it right but it's a major net win on time spent and mental load. It's like having an ultra fast ultra intelligent person pairing with you but you can't talk to each other so it's just kinda guessing.
Most of the time it guesses correctly though it's wrong sometimes. Even for large bits of code it may get 10% of the solution wrong but it generally just takes a minute or two to fix it, where as had I wrote it myself it would take an hour or more.
I prefer to work with developers who don't want to use co-pilot, so I can get things done even faster with less headache and I look even better by comparison.
I've been experimenting with copilot for a few months now and it has been really fun/frustrating. As the post mentions - it's best for fullstack development or learning new languages/toolsets. However, the error rate is still too high for a pure productivity booster, thus it still feels like a fun novelty.
For example, it would help me out with 4 functions and the 5th one would be just plain, confidently wrong and all of those gains are spent on debugging the error. I noticed this especially sucky on lower-level languages or more niche libraries. I was learning Go and it just kept producing errors, introducing unused imports etc. while in Python it can straight up complete my JSON parser predicting the keys and everything.
I wish it had some sort of verbosity setting - this would greatly help with trust. If it outputs some code with single character variables, no comments and some magic comprehension and I need to verify all that? I'm not bothering.
I felt like it was suggesting too often in the beginning, like when it first came out, but now it feels like it's just about perfect as far as when to suggest.
The only negative (and it's super annoying) is when I'm doing an import statement. Normally VSCode is on the money with what files are in the directory, etc, but VSCode has a tendency to make suggestions and generally screw up how it works.
Other than that it's a huge net win even for languages I work on all the time. Especially tedious stuff. I do agree it would be nice to see a ability to throttle how often it makes suggestions though.
> Using GitHub Copilot feels like I’m pairing with a seasoned developer
It feels like pairing with an extremely eager junior developer who constantly makes horrific and subtle mistakes.
It's great at repeating the same pattern or generating boilerplate. But it has no idea how your code works at any scale. So it easily violates invariants, makes up fields, gets confused about variables, etc. Off by one errors are just an endless mess.
As a snippets library, it's awesome. As a pair programming buddy, it's like programming with a psychopath that wants your code to fail at all costs.
As a pair programming buddy, it's like programming with a psychopath that wants your code to fail at all costs.
A good summary though I have to say I find this sort of endearing. When I'm trying something completely new the most frustrating part is not knowing what my options are, eg due to unfamiliarity with a library's classes or (thinking back to when I was first learning a language) how nested data structures like lists of dicts worked, so that I couldn't guess how I was supposed to address them. While a code assistant usually crashes in more or less Wile E Coyote ways, watching it launch confidently is often helpful. It's sometimes easier to fix something that doesn't work than to know the first thing to try.
As with ChatGPT, I've found that Copilot is actually deceptively difficult to master.
There's absolutely a knack to it, and the more time you spend with it the better you get at prompting it for the kind of completions you want.
I'm at a point now where I can often predict what Copilot is going to write for me - which is a HUGE productivity booster, because it means I can effectively use it as a super-fast typing assistant.
I actually do quite a lot of writing in it, by switching VS Code to Markdown mode. If I'm writing documentation I'll paste the code I'm documenting into the markdown file, write the Copilot-assisted docs and then delete the code when I'm finished.
I've been using it full-time for about two months now. On balance I've found CoPilot useful and plan to keep the paid subscription beyond the trial. But one thing I've noticed is that it's easy to get lazy and let it generate variations on stuff for you that really should be refactored out into a reusable class/component/whatever. It's great for rapid prototyping because it can save you a lot of round trips to Stack Overflow or API docs, but it quickly leads you in the direction of way too much duplication. That's not surprising since today it's a write-only generative tool.
I wish the latency were better. I find myself waiting on it sometimes which kinda breaks the magic and flow of using it.
It doesn't happen all the time, but there have definitely been some moments where I'm astounded at what it anticipates & generates. I'm hoping it will do this more often as the tech matures.
It’s not going to be write-only for long: the Copilot Labs version of the VS Code extension introduces brushes, which (by analogy to Photoshop) can be used to retouch your code [1]. Make code more readable, fix bugs, add types, and so on. Exciting times. I’m already regularly blown away by what it generates; the idea that it could similarly ‘read my mind’ with the refactoring In want to do seems just round the corner.
There is no doubt some additional value that copilot or code whisperer brings , though opinions varies wildly. Denying that or denying the tools to developers as a policy will only hurt everyone. I think it is best left to individuals to decide for themselves.
For me, Copilot is like a smart snippets library. But like a snippets library, don't ever expect it to do something novel. It's great for boilerplate, documentation, obvious blocks of code, etc., but if there's any nuance or novelty, I've found I do best turning it off for a while.
Like others said, it also leads to a lot of code if you're lazy. Keep refactoring!
Thats exactly how I described it to coworkers recently. Its like having a junior developer at your disposal that does all the boring and monotonous stuff. It's not doing anything crazy and new, that's why you still have a job.
I'll admit I don't leverage this savings into any massive productivity benefits that my CTO is going to notice, but it does significantly reduce my fatigue throughout the day which keeps me sharper, happier, and more energetic later into the day.
It keeps suggesting code that doesn't work, like when I call three different setters it will suggest a setter that looks like it might fit but when I complete the suggestion it's a setter that doesn't exist. I find it very distracting and so far am unimpressed.
It's pretty good at writing tests, which makes sense really, as it can read the code and interpret what you want based other code and for a statically typed language, know what a function should return and it should asset for.
That's all I use it for, hashing out tests, and auto-completion of structs where is seems to figure out what I need quite well. For my main code , it's more of a case of 'go home co-pilot, you're drunk'.
So, so often I accept a snippet from Copilot, only to find in like, the next 5 minutes, that it got something a bit wronger than I had hoped it'd save me (e.g. looking up a variable name or what the API requires, etc.).
It's fine, I use it and it's worth the $10, but it's not a gamechanger. It blew me away at first, just like the author, but nowadays it's just a normal part of my work day.
Are alot of the problems people complaining about due to dynamic languages like JavaScript where it's not immediately known if a property exists or not?
Using C#, Copilot will often preemptively assume I want a property or method that does not yet exist, but obviously in Rider/VS it displays instantly that it doesn't exist and can be generated right away. So some of the things people complain about here where it uses variables/properties/methods that don't exist aren't problems I'm dealing with.
I’ve been using copilot for 2 months. I’ve past the free period. I decided I would conti ue to pay for it. Last week I found out my company would pay for it to!
My stack it TS, Vue, Nuxt app.
Copilot is great for tests. I write the assertion and copilot completes the mocked data unit testing the function.
Copilot is also really good and large tedious refactors. Updating some dependency’s with breaking change and you have to do 100+ file change in a similar way. After you have the new pattern figured out the autocomplete is a massive time saver.
Also saves times on making types!
Cons
Declares vars, properties names wrong. Ex if you have 2 properties in an object (isOpen, isFinished) and you are making some type for it. T
he autocomplete can be wrong.
It would suggest isOpen and isClosed instead of following the names provided. Seems like a minor naming pattern that is weighted over other inferences.
So the con is that while writing code you would need to check variable names, properties are declared correctly. Overall it’s small effort that I might have to do anyway during a development.
I also have dyslexia so any linting, TS checking and now copilot takes burden off my error prone writing
Copilot is a horrible crutch. I will never use it, because I know how to read the documentation/source and write my own code. It makes me sad to see it embraced, as I see it as nothing more than a harbinger of legions of even worse software than we already must tolerate.
I am stunned at how people think it okay to use in a professional or educational setting, when the general reaction to ChatGPT in those settings has been mostly negative. Both are leading the race to the bottom, producing mountains of crap output that will diminish the quality of the internet until it is unusable.
I'd agree with you if using Copilot meant using and never validating the code that's suggested. Adding it to one's toolbelt can mean coming up with the syntax to then consult the documentation with any questions. But often I imagine it'd just spit out boilerplate code that looks very familiar anyway. I can't come up with the right syntax for various http libraries but I recognize the right one, I just don't want to write it out.
Is that really faster than just reading the docs and editing some example code? You have to prompt an AI for what you need, have it generate the code for you, then read the code and validate it. That does not seem like a labor saving process to me.
> Why waste brain cells and time on mundane things when my time and attention is better spent thinking about the problem at hand.
This just sounds like a thought-terminating cliché; what mundane things are you spending your attention on while writing code? Sure, there are times where you might be generating boilerplate, cool, you're paying for a fancy snippet library, except the snippets are potentially malformed so you should check them every time.
> what mundane things are you spending your attention on while writing code?
To stick with my http library example, figuring out how to pass headers or a json body to a particular library, checking status codes, was it "status_code" or just status, is the body in an attribute called body or content or json or json(). I know this is trivial if you work with the same library over and over again but people are different and some of us work with a wide array of languages or libraries.
You can stay in your editor, even in the same file. That for sure saves some time. You have to type out the function name and a comment probably anyway. I don't see how this couldn't be a net positive in terms of saved time and reduced context switching but we'll see.
You haven't made any kind of argument, but I can fathom from your tone about 'ChatGPT' that there is something inherently wrong with generated code.
I suggest you're implied reasoning is wrong: if it's a tool and it helps, use it.
I don't use it because the code I write is hyper specific, and the suggestions interfere with the 'regular' code completions, and they're just not good enough.
But it is very impressive, and for boiler-plate oriented code, I would use it.
All of that said, the risk in the code is that it can introduce a bit of laziness in that code still needs to be very closely inspected and I suggest many won't bother.
> All of that said, the risk in the code is that it can introduce a bit of laziness in that code still needs to be very closely inspected and I suggest many won't bother.
I think this is the crux of the problem for such "generated code". It's like when you do a code review of a pull request - it's very easy to miss subtle issues. When you write the code yourself you're forced to really think things through. You will still make mistakes, granted.
But accepting generated code that you "reviewed" is like you're back in math class convincing yourself that you understood the teacher's line of reasoning and you know the material. It's when you really get down to business with your nose on the facts when you will see subtle (and not so subtle) bits where either copilot got it wrong or where you simply didn't understand some implications.
> Copilot is a horrible crutch. I will never use it, because I know how to read the documentation/source and write my own code.
I think it depends on how you use it. I never let Copilot make design decisions, because it kind of sucks at that, and it that carries a lot of risk of wasting more time than it saves in the long run.
But there are some tasks that I think it's well suited for. In a recent commit[1] to one of my side projects, I swapped out one graphics library for another, which involved replacing a bunch of library calls and would have been tedious and boring to write by hand. I turned Copilot on for this, and after I did a couple substitutions by hand it was able to do most of the work and was easily a net positive.
For my own usage, I keep copilot off in emacs, and opt to flip it on with a keybinding if and when I know exactly what the code I want to be written should look like, but actually writing it out would be a pain.
> I will never use it, because I know how to read the documentation/source and write my own code.
I will never use a file system because I know how to store my own punchcards!
I will never use a compiler because I know how to right my own assembly.
I will never use an IDE because I know how to edit code with SED.
I will never use a debugger because I know how to debug with logging statements.
I will never… use new tools because I learned the old ones?
I agree there’s some open questions about the copyright issues which should give people pause before use in some context. That said, it’s a tool that helps people work faster and focus on higher priority issues than implementing trivial code. Oh and a lot of big software companies are already developing this sort of tool internally so it’s going to be the norm for competitive purposes.
Do you also require code be written on notepad.exe on a monochrome display or am I allowed to use autocompleting IDEs? Where do you draw the line on computer assisted text editing?
I still have trouble imagining a software developer saying "enhances my experience as a developer" (which still sounds to me like something a bad marketing person from decades ago might say, if they didn't understand software developers).
Which is maybe related to why I also have trouble understanding all these people voluntarily making written statements that their work product is tainted by Copilot, before the copyright issues have been settled in the courts.
And, in this particular instance, doing it on the blog of what seems to be a contract software development firm serving other companies, they'd potentially be prompting the question of whether their tainted work product exposed their customers and/or violated contracts with their customers.
This entire article felt generic enough that for a moment I thought that they'd generated it using GPT. It's a depressing state of affairs when this starts to become the norm.
I'm using Copilot as well. And it tells a lot because I don't like to pay for software (right now my only subscriptions are Idea and Copilot). I think that it has a lot of room for improvement. For example right now it only adds code but updating code would be very useful as well. Also finding subtle bugs would be very useful too and I'm sure that Copilot can do that. But even in its current form it allows to offload some boring code generation to AI. It's like auto-complete, but on the next level.
50 comments
[ 3.3 ms ] story [ 94.5 ms ] threadThere are times that it doesn't get it right but it's a major net win on time spent and mental load. It's like having an ultra fast ultra intelligent person pairing with you but you can't talk to each other so it's just kinda guessing.
Most of the time it guesses correctly though it's wrong sometimes. Even for large bits of code it may get 10% of the solution wrong but it generally just takes a minute or two to fix it, where as had I wrote it myself it would take an hour or more.
I prefer to work with developers who don't want to use co-pilot, so I can get things done even faster with less headache and I look even better by comparison.
For example, it would help me out with 4 functions and the 5th one would be just plain, confidently wrong and all of those gains are spent on debugging the error. I noticed this especially sucky on lower-level languages or more niche libraries. I was learning Go and it just kept producing errors, introducing unused imports etc. while in Python it can straight up complete my JSON parser predicting the keys and everything.
I wish it had some sort of verbosity setting - this would greatly help with trust. If it outputs some code with single character variables, no comments and some magic comprehension and I need to verify all that? I'm not bothering.
The only negative (and it's super annoying) is when I'm doing an import statement. Normally VSCode is on the money with what files are in the directory, etc, but VSCode has a tendency to make suggestions and generally screw up how it works.
Other than that it's a huge net win even for languages I work on all the time. Especially tedious stuff. I do agree it would be nice to see a ability to throttle how often it makes suggestions though.
It feels like pairing with an extremely eager junior developer who constantly makes horrific and subtle mistakes.
It's great at repeating the same pattern or generating boilerplate. But it has no idea how your code works at any scale. So it easily violates invariants, makes up fields, gets confused about variables, etc. Off by one errors are just an endless mess.
As a snippets library, it's awesome. As a pair programming buddy, it's like programming with a psychopath that wants your code to fail at all costs.
A good summary though I have to say I find this sort of endearing. When I'm trying something completely new the most frustrating part is not knowing what my options are, eg due to unfamiliarity with a library's classes or (thinking back to when I was first learning a language) how nested data structures like lists of dicts worked, so that I couldn't guess how I was supposed to address them. While a code assistant usually crashes in more or less Wile E Coyote ways, watching it launch confidently is often helpful. It's sometimes easier to fix something that doesn't work than to know the first thing to try.
There's absolutely a knack to it, and the more time you spend with it the better you get at prompting it for the kind of completions you want.
I'm at a point now where I can often predict what Copilot is going to write for me - which is a HUGE productivity booster, because it means I can effectively use it as a super-fast typing assistant.
I actually do quite a lot of writing in it, by switching VS Code to Markdown mode. If I'm writing documentation I'll paste the code I'm documenting into the markdown file, write the Copilot-assisted docs and then delete the code when I'm finished.
I wrote up a few tips here: https://til.simonwillison.net/tils/search?q=Copilot
I wish the latency were better. I find myself waiting on it sometimes which kinda breaks the magic and flow of using it.
It doesn't happen all the time, but there have definitely been some moments where I'm astounded at what it anticipates & generates. I'm hoping it will do this more often as the tech matures.
[1]: https://githubnext.com/projects/code-brushes
Like others said, it also leads to a lot of code if you're lazy. Keep refactoring!
Sometimes it's not but it is enough to be very valuable.
I'll admit I don't leverage this savings into any massive productivity benefits that my CTO is going to notice, but it does significantly reduce my fatigue throughout the day which keeps me sharper, happier, and more energetic later into the day.
That's all I use it for, hashing out tests, and auto-completion of structs where is seems to figure out what I need quite well. For my main code , it's more of a case of 'go home co-pilot, you're drunk'.
It's fine, I use it and it's worth the $10, but it's not a gamechanger. It blew me away at first, just like the author, but nowadays it's just a normal part of my work day.
Using C#, Copilot will often preemptively assume I want a property or method that does not yet exist, but obviously in Rider/VS it displays instantly that it doesn't exist and can be generated right away. So some of the things people complain about here where it uses variables/properties/methods that don't exist aren't problems I'm dealing with.
My stack it TS, Vue, Nuxt app. Copilot is great for tests. I write the assertion and copilot completes the mocked data unit testing the function.
Copilot is also really good and large tedious refactors. Updating some dependency’s with breaking change and you have to do 100+ file change in a similar way. After you have the new pattern figured out the autocomplete is a massive time saver.
Also saves times on making types!
Cons Declares vars, properties names wrong. Ex if you have 2 properties in an object (isOpen, isFinished) and you are making some type for it. T he autocomplete can be wrong.
It would suggest isOpen and isClosed instead of following the names provided. Seems like a minor naming pattern that is weighted over other inferences.
So the con is that while writing code you would need to check variable names, properties are declared correctly. Overall it’s small effort that I might have to do anyway during a development.
I also have dyslexia so any linting, TS checking and now copilot takes burden off my error prone writing
I am stunned at how people think it okay to use in a professional or educational setting, when the general reaction to ChatGPT in those settings has been mostly negative. Both are leading the race to the bottom, producing mountains of crap output that will diminish the quality of the internet until it is unusable.
I'd agree with you if using Copilot meant using and never validating the code that's suggested. Adding it to one's toolbelt can mean coming up with the syntax to then consult the documentation with any questions. But often I imagine it'd just spit out boilerplate code that looks very familiar anyway. I can't come up with the right syntax for various http libraries but I recognize the right one, I just don't want to write it out.
I am using another code completion AI in my IDE, and it’s wonderful.
Why waste brain cells and time on mundane things when my time and attention is better spent thinking about the problem at hand.
And in the future, AI will be able to take over more and more of the time. Eventually we will not be needed at all.
This just sounds like a thought-terminating cliché; what mundane things are you spending your attention on while writing code? Sure, there are times where you might be generating boilerplate, cool, you're paying for a fancy snippet library, except the snippets are potentially malformed so you should check them every time.
To stick with my http library example, figuring out how to pass headers or a json body to a particular library, checking status codes, was it "status_code" or just status, is the body in an attribute called body or content or json or json(). I know this is trivial if you work with the same library over and over again but people are different and some of us work with a wide array of languages or libraries.
I code in Rust but said AI code completion works with some other languages also
https://plugins.jetbrains.com/plugin/12798-tabnine-ai-code-c...
I suggest you're implied reasoning is wrong: if it's a tool and it helps, use it.
I don't use it because the code I write is hyper specific, and the suggestions interfere with the 'regular' code completions, and they're just not good enough.
But it is very impressive, and for boiler-plate oriented code, I would use it.
All of that said, the risk in the code is that it can introduce a bit of laziness in that code still needs to be very closely inspected and I suggest many won't bother.
I think this is the crux of the problem for such "generated code". It's like when you do a code review of a pull request - it's very easy to miss subtle issues. When you write the code yourself you're forced to really think things through. You will still make mistakes, granted.
But accepting generated code that you "reviewed" is like you're back in math class convincing yourself that you understood the teacher's line of reasoning and you know the material. It's when you really get down to business with your nose on the facts when you will see subtle (and not so subtle) bits where either copilot got it wrong or where you simply didn't understand some implications.
I think it depends on how you use it. I never let Copilot make design decisions, because it kind of sucks at that, and it that carries a lot of risk of wasting more time than it saves in the long run.
But there are some tasks that I think it's well suited for. In a recent commit[1] to one of my side projects, I swapped out one graphics library for another, which involved replacing a bunch of library calls and would have been tedious and boring to write by hand. I turned Copilot on for this, and after I did a couple substitutions by hand it was able to do most of the work and was easily a net positive.
For my own usage, I keep copilot off in emacs, and opt to flip it on with a keybinding if and when I know exactly what the code I want to be written should look like, but actually writing it out would be a pain.
[1] https://github.com/jpe90/model-view-projection/commit/1ed4fb...
The only trouble is, at the moment it makes subtle mistakes. So you really have to know what you're doing to use it properly.
I reckon we're 6 months away from those mistakes becoming few and far between.
I will never use a file system because I know how to store my own punchcards!
I will never use a compiler because I know how to right my own assembly.
I will never use an IDE because I know how to edit code with SED.
I will never use a debugger because I know how to debug with logging statements.
I will never… use new tools because I learned the old ones?
I agree there’s some open questions about the copyright issues which should give people pause before use in some context. That said, it’s a tool that helps people work faster and focus on higher priority issues than implementing trivial code. Oh and a lot of big software companies are already developing this sort of tool internally so it’s going to be the norm for competitive purposes.
Which is maybe related to why I also have trouble understanding all these people voluntarily making written statements that their work product is tainted by Copilot, before the copyright issues have been settled in the courts.
And, in this particular instance, doing it on the blog of what seems to be a contract software development firm serving other companies, they'd potentially be prompting the question of whether their tainted work product exposed their customers and/or violated contracts with their customers.