ChatGPT is instrumental for me to maintain pyspark code efficiently. I don't want to learn it as I'm not particularly interested in the projects that use it. It's a lot easier to just learn on the fly and double check.
It's better than traditional autocomplete, but not transformative. ChatGPT is particularly bad with JavaScript — you're better off writing that yourself.
Thing is, Copilot doesn't seem to take the code of the actual installed packets into account.
The moment you work with something a bit more obscure, it just guesses.
Especially with TypeScript it's kinda funny, since IntelliSense which is right there, in the IDE, tells the truth, but is completely ignored by Copilot.
I had an interesting experience with CoPilot yesterday. I asked it to add a 'deactivate' button to each of a collection of items. It did that, but unexpectedly, it conditionally displayed 'reactivate' if the `item.deactivated_at.present?` with appropriate bootstrap icons (a cross for deactivate and a circular arrow for reactivate).
What surprised me was it knew what I wanted better than I did (for this MVP, I hadn't even considered 'reactivate' functionality - I'd have been happy just with deactivate).
So it didn't just write code I asked for, but suggested (and was right about) something it thought I might like beyond what was actually asked for.
It was my first time using CoPilot chat so perhaps it does this a lot, but it was my first time experiencing it. It challenged my thinking and improved it.
How exactly had it improved your thinking? By your description it did it for you, you just sat back.
Doing nothing does not drive self improvement, I can't get stronger sitting on the couch watching videos on YouTube about weight lifting, even if I did type in the search term.
Who cares about developing skills machines can do better. I don’t spend hours with a handsaw so I can cut perfect perpendicular cuts in wood, I just use a miter saw that does it perfectly 99% of the time. That lets me concentrate on the next thing on the value chain that machines can’t do yet.
There’s plenty of woodworkers that work exclusively with hand tools for the joy of it and that where a lot of programming will be a decade in the future.
Because well, for me we are designing tools not doing manual labor. The tool is designed once and well. Then the computer executes it 50 million times. Automated tool design is cool but the ROI for us is not developer hours minimized but number of users.
The software I make is the tool. The user uses my tool millions of times. The cost of adding a user is close to zero. So the power for me isn't in reducing developer costs
My father was a wiz with a slide rule and understood the relations between logarithms and powers as a matter of necessity - he could look at a number and have a rough idea of what the natural log of that value was at a glance. I had a calculator - I am an expert at pushing the buttons but outside of a few select important numbers if you throw a value at me I'm not going to guess correctly - log10(7000) is clearly between 3 and 4 (it's in the thousands) - is it closer to 3.7, 3.5 or 3.9?
Tools don't improve skills they replace but, at the same time, are those skills that are being replaced important to practice? If you've worked even briefly with vector math or game programming you know sqrt(2) is 1.41 being able to summon that value at will is essential if you're regularly doing vector addition - but sqrt(23)? Probably never going to come up.
At the end of the day we need to decide whether these tools are helpful or not - in the very limited scope of UI building I think the answer is yes... a lot of front-end surface-level development has one right answer. For other things I am skeptical - and even for front-end work a developer that is considerate with their CSS classes and tag nesting will be a darling of the UI team compared to a developer that gets it to look right but does so with code that's difficult to understand or modify.
Are you suggesting that you wouldn't want anyone (or anything) to suggest improvements for you because then you would not learn yourself how to improve?
By "It challenged my thinking and improved it." OP probably meant 'improved upon it', or 'improved on it' where "it" was the originally desired output.
Even in not refined software: the other day I asked it to, in js, create a table taking (hour, power produced) data from a json response. It managed to come up with an n^2 way of doing what should have been a linear traversal through the json.
Sometimes it's great, sometimes the middle of the curve is itself bad. :)
Great point; sometimes average is not good enough. (imagine what could go wrong with a banking application or software for dangerous physical/industrial process). A simple inefficiency like the that could cause degradation in a safety system or similar.
This strengthens the case for careful assessment of when CoPilot is used vs not used, and in cases that it's used, how much effort goes into rigorously testing and scrutinizing its suggestions (as opposed to unthinkingly accepting them as one may for tasks known to have no downside risk).
Maybe there is an option to make Copilot prioritize security concern over other aspects when working in the context of security related apps or features.
This is how humans work, too.
Technically, this is fine tuning or prompt engineering.
It feels important to me, from the perspective of someone who doesn't want to be negatively surprised by auto-generated code. I'd expect there to be a significant difference in uncaught errors in Copilot code between people who believe Copilot knows what they want vs people who think copilot just does what it's been trained to do, mainly based on the level of scepticism people approach reviewing the code with.
I think it is a huge difference. It didn't know what you wanted. It knew what most people want.
For a lot of generic things what you want is probably what most people want. Shoot, most times we ask ourselves what other people do in this situation.
But in a specific case, what you want is probably not what other people want. As that is why you are writing this code, to do something different.
I have the same experiences, but i am using already for a year. Quite surprised now and then how its able to predict my next move. Sometimes remarkably accurate. I noticed here on hackernews there is a large group of copilot haters. I normally find this forum
a place with the best open minded people out there. But not with copilot topics :)
Being open minded doesn't imply one has to accept incorrect terms like "knows" and "predicts" when talking about technical topics involving LLMs. HN of all communities should do better.
> While suggestion correctness is important, the driving factor for these improvements appears to be not correctness as such, but whether the suggestions are useful as a starting point for further development.
I admit I didn't read the whole article, but that bullet I thought was key. I totally agree. When folks yell "BuT HalliciNations!!!", I get it, but that doesn't mean that LLMs can still be a huge boon if you know how to use them and don't just trust their output blindly and yolo it into production.
Precisely. I'm all in favour of AI/LLM/GAN/Whatevers being used as _an_ input to the process of human creativity or judgement. They only become troublesome when their output is blindly trusted.
I love Copilot, I find it essential these days, but if it's going to significantly impact my productivity it's got to be 2x faster. The autocompete is scary good at predicting what was in my head, sometimes before I figure it out, but often it trails behind my own speed.
You might be right for small stuff, but I’ve done extensive refactoring with it and anything beyond a single line completion takes noticeable time gaps.
There was a Show HN a few days ago [0] from a service promising ~3x faster copilot-style autocomplete. I can confirm that it is extremely fast (sometimes too fast - I use ctrl+right arrow a lot to navigate, which with most completion services isn't a problem because I'll have moved the cursor before a completion comes back, but with supermaven I've been accidentally accepting the completion instead).
I think its most of all extremely situational. For repetive simple tasks, you can produce code at probably something like 50x the rate, but for anything nontrivial it's at best useless and more likely a distracting nuisance.
So the speedup is likely a function of what you are doing. Application logic, high. Lower level stuff, low.
The outcome they are measuring is "perceived productivity", which seems pretty weak sauce to me.
Here, we investigate whether usage measurements of developer interactions with GitHub Copilot can predict perceived productivity as reported by developers.
And they use the "actual activity" to predict this perceived productivity.
To be fair, measuring software reliability / worker productivity like we do other industrial products was an intractably difficult problem long before Copilot showed up :)
I'm already an experienced developer, but writing code in new domains is sooo much nicer now. I can learn and get things done way faster than ever before. Copilot has pretty much replaced Google and Stack Overflow for me. I use it all day everyday. The chat feature is great at well to discuss code, questions, ideas, etc.. I still use ChatGPT 4 for bigger questions, more complex things, writing entire files, etc..
Just built an ncurses python application in very little time, without knowing anything about ncurses at all
Some iterations can be frustrating and it can’t handle everything, but it saves a ton of time in getting to a productive rhythm with tools you hadn’t used before, it kinda flattens the initial learning curve, which is very nice
Copilot has been more impactful for 'unproductive' stuff for me. Not as good at handling the code related to my job, as it's a math heavy beast in an extremely niche field. But great for my hobby, writing little bots to toy around with various things. Lets me skip remembering or looking up a lot of basic implementation details of talking to the service APIs, drastically speeding up the time to get a functional prototype.
As others have mentioned, it's great for speeding up all the little boilerplate and other things too simple or otherwise too unrelated to the main goal.
Sometimes it almost works like rubberduck-development, by seeing what Copilot spits out, I have sometimes realized earlier than otherwise, that I missed certain components/checks that I should probably plan ahead for (eg remembering to add a convenient way to handle user profiles for a bot).
For me, Figure 6 is the most interesting observation: programmers are interacting with Copilot in a statistically different way nights and weekends vs working hours.
I'm guessing (without hard evidence) that this implies day job code-reviewed commits and weekend hobby project/side hustle/startup coding are held to different standards by the developers involved.
Or if doing your day job during off hours, you're just looking for a quick solution and don't actually care about polish because you just want to get back to not working.
I use it for doing mundane tasks like creating queries following a pattern I defined or creating docs for code. My experience has been great so far. Not sure if it is worth the price for my company, but it encourages me being a little lazy and saves some time.
I think it's too early to tell but my main concern about copilot is code maintainability and security. Copilot is able to barf out helpful expressions that will reduce the amount of code we need to write by hand - I think it's excellent when it comes to reducing boilerplate... but I think a large amount of boilerplate existing belies a bigger issue with the project. The majority of software engineering isn't writing code - copilot may be beneficial as an accessibility aide for developers that have typing impairments but most developers can type faster than they can think - if the level of boilerplate in your project is reasonable then this should mean you're never prevented from thinking because your fingers are still working on recording your previous thought. However, at the end of the day, if you can help reduce carpal tunnel that's still a win.
The problem I can foresee with copilot is that the scenario change you're agreeing to is that you'll type less but need to read over the code produced more - this is an effort that isn't normally necessary (typos happen but those should take a trivial time to correct) but when copilot is involved you need to proof all the code that is being generated. There is a motivation to skip this step and just accept the code was written correctly and that will inevitably lead to security problems - and there is a motivation to not correct or alter auto-filled command. If there's a multi-dimensional array and you think it semantically makes sense to iterate it over dimension a then dimension b and copilot instead goes with b as the major index then it's more likely to remain in a b major iteration - that may make code less readable or it may cause major issues down the line.
Copilot, IMO, is optimizing the least important part of development right now and it costs us more to correct it then it would to just splat out the correct code _but_ this is a similar argument to longbows vs. crossbows - hand a peasant a crossbow and they can fire a crossbow - train a peasant for 30 years and they can fire a longbow - the longbow is more powerful, but the crossbow is a clear choice in terms of RoI. It may be that today's developers will only benefit from copilot minimally since we've invested the training time in standard development practices but tomorrow's developers will eschew a lot of the algorithmic learning and still be able to deliver the majority of the value.
> I think it's excellent when it comes to reducing boilerplate
I think the rest of your comment already elaborates this point, but I want to highlight that language models do not reduce boilerplate, they automate it. Reducing boilerplate happens by designing simpler APIs, not by making language models interface with complex APIs for you.
I agree - a lot of my work is actually meta-programming... I've built the view interfaces, the DB handles, the library functions that makes other developer's lives easier - there are real gains you can achieve by making other developers more expressive.
This is so well summarised. Thank you. I also don’t believe the hype.
I tried to migrate spring boot security configuration between major versions and Copilot was just not able to produce a failure free output no matter how long I tried (I tried to hint a lot but it didn’t help).
And in the end software engineering isn’t as much about writing code than to build systems for a certain purpose and in the first place you need to understand the needs/purpose and therefore talk with people instead of just writing code. Writing code is not difficult. Understanding you domain might be.
Pretty sure this bubble will burst and we will have a second wave in years which might be able to do it all. But not this text generation vodoo we see today.
- I have no idea how to use an API, and it is poorly documented, but I am an expert in the general problem it's trying to model
- I need to write code using this API in the next hour
- That code will only ever run 5-6 times, each by hand with an expert to look at any errors or crashes, and then will be thrown away forever
copilot is killer for this scenario. it's the most common scenario when writing machine learning research code. i think probably the people building these tools are enjoying the dogfood, maybe more than the typical end user.
the only thing this llm craze is helping is nvidia/openai/ms war chest, and fueling the illusion that every company can finally have their developers by taking any domain expert or barely knowledgeable person paired with an ai assistant. the mountains of trash produced by these things will end up costing a metric ton.
What I am most looking forward to is future changes in development speed. I wish the report tracked the changes over time. Were they committed as-is (which is a metric to the local quality of the suggestion) and how long does it survive (which is a metric to the global quality of the code).
I’m a generalist who has been good to excellent at various technologies and languages over a long time but honestly am not currently sharp at many. I’ve written Python, TS, SQL and Java, with many API’s and domains in the last few weeks. I don’t remember everything so Copilot and ChatGPT are an excellent way to get me back in the game, although I often have to still identify and fix where it’s going wrong. And some fairly rare (little code online) areas it produces garbage so it’ll take you 10% of the way there and then it’s back to you.
I'd be curious to see improvements in "work being done" metrics. Something like DORA. Where a company has a timeline of metrics before and at some point they introduce Copilot or other types of "AI" assistants. I suppose we will start seeing these, since its something companies would like to share with their shareholders.
For me copilot is mildly helpful 10% of the time when coding, but writing code take up such a small amount of my time that it doesn't make a measurable difference overall.
let's also measure the productivity of reviewers and people in general that, at a later point, have to wade through piles of ai generated crap.
last friday i had to review 2 trash PRs that were blatantly made with ai coding assistance. hundreds of code lines for something that, by reading the doc of the library, could have been made in 5 lines. and the fantastic comments like "returns the body" over a body() function.
N=1 but I find Copilot to be incredibly valuable to me.
So much so that when there was an outage a few weeks ago (or maybe I had network issues on my side) the loss of it was palpable. I found myself pausing, waiting for Copilot to spit out code only to realize it wasn't going to do it. Once you've used Copilot for a while you get a good sense of what it can and can't do. When to pause and when to just keep typing. I was so used to knowing "this is a thing Copilot will do well" and waiting for it that I kept forgetting it wasn't working right now.
Even before this experience I was convinced of the usefulness of it. I've been writing code for close to 20 years and I think I'm pretty decent at it, I never take Copilot's suggestion without first understanding what it's doing but more often than not the suggestion is almost identical to what I would have written myself. Sometimes it wants to do a `.forEach` and I would prefer a `for()` loop but that's easy to fix and often writing `for(` is enough for it to re-write that part of code in the way I prefer. Those changes are often only stylistic.
In addition, it's great for code I don't write often but need something quick and dirty to test out a POC. It along with ChatGPT feel like cheating. Just yesterday we were looking into an issue where I work. We had some timing data in the logs but nothing was consuming/displaying that data. Yes, we could grep for the lines of data but we didn't have this feeding into prometheus and the effort to do that was not going to be minor.
Instead I had ChatGPT parse the log lines I had already filtered with grep and spit out CSV data ("Datetime, how many seconds something took") then I had it write an extremely basic HTML/CSS/JS file to graph the data. After checking that it was all working I hooked up the command directly in the php file that held the graph so we had "live" graphing (after a reload) of a problem we were investigating. This whole thing took well under 5 minutes.
Now I'm perfectly capable of doing everything I just outlined above but it would have taken me longer than 5 minutes just to look up and use the ChartJS syntax/api. Instead I had a tool displaying near-live data in almost no time at all.
"AI" feels like a superpower. I already know what I want to do and often I even know how to write the code to do it but LLMs let me skip the repetitive boring parts and focus on the things LLMs are not good at, my specific problem space, the specifics of my stack, etc. Only I can do that (for now at least), let the LLM spit out graphs, loops, awk commands, etc, I'll glue it all together and make it useful.
Where it really shone while I was trialling it was its ability to guess exactly what throwaway code I wanted while I was prototyping this and that and wrapping it all up in a println! call.
It was extremely handy from an autocomplete perspective - EXCEPT it insisted on inserting triple backticks into my Rust code four out of five times
78 comments
[ 3.2 ms ] story [ 149 ms ] threadThe moment you work with something a bit more obscure, it just guesses.
Especially with TypeScript it's kinda funny, since IntelliSense which is right there, in the IDE, tells the truth, but is completely ignored by Copilot.
It feels like how auto complete should actually work, but I haven't seen any exceptional performance from Copilot.
Like with Phind, the longer the code examples, the more errors.
But it feels a more fluid when typing boilerplate heavy code, which is nice.
Con: writing lots of boilerplate is less painful.
(So you get more boilerplate code)
What surprised me was it knew what I wanted better than I did (for this MVP, I hadn't even considered 'reactivate' functionality - I'd have been happy just with deactivate).
So it didn't just write code I asked for, but suggested (and was right about) something it thought I might like beyond what was actually asked for.
It was my first time using CoPilot chat so perhaps it does this a lot, but it was my first time experiencing it. It challenged my thinking and improved it.
Doing nothing does not drive self improvement, I can't get stronger sitting on the couch watching videos on YouTube about weight lifting, even if I did type in the search term.
There’s plenty of woodworkers that work exclusively with hand tools for the joy of it and that where a lot of programming will be a decade in the future.
It's not like watching others lift, it's like having a sparring partner. ( A very fast, slightly stupid sparring partner...)
Coding a button isn't the part of software development that's intellectually stimulating, so I am glad we have robots to help us speed it up.
I feel like you're coming from a place of disliking chatbots and then working backwards.
Tools don't improve skills they replace but, at the same time, are those skills that are being replaced important to practice? If you've worked even briefly with vector math or game programming you know sqrt(2) is 1.41 being able to summon that value at will is essential if you're regularly doing vector addition - but sqrt(23)? Probably never going to come up.
At the end of the day we need to decide whether these tools are helpful or not - in the very limited scope of UI building I think the answer is yes... a lot of front-end surface-level development has one right answer. For other things I am skeptical - and even for front-end work a developer that is considerate with their CSS classes and tag nesting will be a darling of the UI team compared to a developer that gets it to look right but does so with code that's difficult to understand or modify.
Are you suggesting that you wouldn't want anyone (or anything) to suggest improvements for you because then you would not learn yourself how to improve?
You can call productive if you want, but sounds pretty far from self development to me.
Learning is easy. It's just learning.
If CoPilot bumps these crumby solutions into 'average' solutions then it's a big improvement under those circumstances.
But when making highly refined software, CoPilot's suggestions might be relatively mundane; a degradation in quality.
Sometimes it's great, sometimes the middle of the curve is itself bad. :)
This strengthens the case for careful assessment of when CoPilot is used vs not used, and in cases that it's used, how much effort goes into rigorously testing and scrutinizing its suggestions (as opposed to unthinkingly accepting them as one may for tasks known to have no downside risk).
Maybe there is an option to make Copilot prioritize security concern over other aspects when working in the context of security related apps or features.
This is how humans work, too.
Technically, this is fine tuning or prompt engineering.
I personally say "it" as I see it as a machine.
> Quite surprised now and then how its able to predict my next move
Not text, the user's next move.
I admit I didn't read the whole article, but that bullet I thought was key. I totally agree. When folks yell "BuT HalliciNations!!!", I get it, but that doesn't mean that LLMs can still be a huge boon if you know how to use them and don't just trust their output blindly and yolo it into production.
But anyway, the future is exciting.
[0] https://news.ycombinator.com/item?id=39473773
So the speedup is likely a function of what you are doing. Application logic, high. Lower level stuff, low.
Here, we investigate whether usage measurements of developer interactions with GitHub Copilot can predict perceived productivity as reported by developers.
And they use the "actual activity" to predict this perceived productivity.
Just built an ncurses python application in very little time, without knowing anything about ncurses at all
Some iterations can be frustrating and it can’t handle everything, but it saves a ton of time in getting to a productive rhythm with tools you hadn’t used before, it kinda flattens the initial learning curve, which is very nice
As others have mentioned, it's great for speeding up all the little boilerplate and other things too simple or otherwise too unrelated to the main goal.
Sometimes it almost works like rubberduck-development, by seeing what Copilot spits out, I have sometimes realized earlier than otherwise, that I missed certain components/checks that I should probably plan ahead for (eg remembering to add a convenient way to handle user profiles for a bot).
I'm guessing (without hard evidence) that this implies day job code-reviewed commits and weekend hobby project/side hustle/startup coding are held to different standards by the developers involved.
The problem I can foresee with copilot is that the scenario change you're agreeing to is that you'll type less but need to read over the code produced more - this is an effort that isn't normally necessary (typos happen but those should take a trivial time to correct) but when copilot is involved you need to proof all the code that is being generated. There is a motivation to skip this step and just accept the code was written correctly and that will inevitably lead to security problems - and there is a motivation to not correct or alter auto-filled command. If there's a multi-dimensional array and you think it semantically makes sense to iterate it over dimension a then dimension b and copilot instead goes with b as the major index then it's more likely to remain in a b major iteration - that may make code less readable or it may cause major issues down the line.
Copilot, IMO, is optimizing the least important part of development right now and it costs us more to correct it then it would to just splat out the correct code _but_ this is a similar argument to longbows vs. crossbows - hand a peasant a crossbow and they can fire a crossbow - train a peasant for 30 years and they can fire a longbow - the longbow is more powerful, but the crossbow is a clear choice in terms of RoI. It may be that today's developers will only benefit from copilot minimally since we've invested the training time in standard development practices but tomorrow's developers will eschew a lot of the algorithmic learning and still be able to deliver the majority of the value.
I think the rest of your comment already elaborates this point, but I want to highlight that language models do not reduce boilerplate, they automate it. Reducing boilerplate happens by designing simpler APIs, not by making language models interface with complex APIs for you.
I tried to migrate spring boot security configuration between major versions and Copilot was just not able to produce a failure free output no matter how long I tried (I tried to hint a lot but it didn’t help).
And in the end software engineering isn’t as much about writing code than to build systems for a certain purpose and in the first place you need to understand the needs/purpose and therefore talk with people instead of just writing code. Writing code is not difficult. Understanding you domain might be.
Pretty sure this bubble will burst and we will have a second wave in years which might be able to do it all. But not this text generation vodoo we see today.
- I have no idea how to use an API, and it is poorly documented, but I am an expert in the general problem it's trying to model
- I need to write code using this API in the next hour
- That code will only ever run 5-6 times, each by hand with an expert to look at any errors or crashes, and then will be thrown away forever
copilot is killer for this scenario. it's the most common scenario when writing machine learning research code. i think probably the people building these tools are enjoying the dogfood, maybe more than the typical end user.
Not perfect but invaluable.
last friday i had to review 2 trash PRs that were blatantly made with ai coding assistance. hundreds of code lines for something that, by reading the doc of the library, could have been made in 5 lines. and the fantastic comments like "returns the body" over a body() function.
So much so that when there was an outage a few weeks ago (or maybe I had network issues on my side) the loss of it was palpable. I found myself pausing, waiting for Copilot to spit out code only to realize it wasn't going to do it. Once you've used Copilot for a while you get a good sense of what it can and can't do. When to pause and when to just keep typing. I was so used to knowing "this is a thing Copilot will do well" and waiting for it that I kept forgetting it wasn't working right now.
Even before this experience I was convinced of the usefulness of it. I've been writing code for close to 20 years and I think I'm pretty decent at it, I never take Copilot's suggestion without first understanding what it's doing but more often than not the suggestion is almost identical to what I would have written myself. Sometimes it wants to do a `.forEach` and I would prefer a `for()` loop but that's easy to fix and often writing `for(` is enough for it to re-write that part of code in the way I prefer. Those changes are often only stylistic.
In addition, it's great for code I don't write often but need something quick and dirty to test out a POC. It along with ChatGPT feel like cheating. Just yesterday we were looking into an issue where I work. We had some timing data in the logs but nothing was consuming/displaying that data. Yes, we could grep for the lines of data but we didn't have this feeding into prometheus and the effort to do that was not going to be minor.
Instead I had ChatGPT parse the log lines I had already filtered with grep and spit out CSV data ("Datetime, how many seconds something took") then I had it write an extremely basic HTML/CSS/JS file to graph the data. After checking that it was all working I hooked up the command directly in the php file that held the graph so we had "live" graphing (after a reload) of a problem we were investigating. This whole thing took well under 5 minutes.
Now I'm perfectly capable of doing everything I just outlined above but it would have taken me longer than 5 minutes just to look up and use the ChartJS syntax/api. Instead I had a tool displaying near-live data in almost no time at all.
"AI" feels like a superpower. I already know what I want to do and often I even know how to write the code to do it but LLMs let me skip the repetitive boring parts and focus on the things LLMs are not good at, my specific problem space, the specifics of my stack, etc. Only I can do that (for now at least), let the LLM spit out graphs, loops, awk commands, etc, I'll glue it all together and make it useful.
It was extremely handy from an autocomplete perspective - EXCEPT it insisted on inserting triple backticks into my Rust code four out of five times
Here is the art I found about GitHub Copilot: https://gitprotect.io/blog/github-copilot-introduction-an-ai... .