Especially since the article has virtually nothing to do with that fanfic besides "I'm trying stuff to see what happens". The author isn't even following the the scientific method, he's not doing any predictions, just poking the thing and seeing how it reacts.
(Which, to be clear, is perfectly fine and useful, but it's weird that the author treats his process with so much reverence when he's not actually following the formal parts.)
Copilot is at its best (currently) when you teat it as autocomplete on steroids, it's very good at repetitive tasks, predicting the next line, has a brilliant short term memory, and great at prompting "wiring" type tasks.
ChatGPT is far better at this sort of "describe what you want and it will code for you" process from TFA. It looks like the next version of Copilot with the chat interface will bring the latter workflow into the system, I'm looking forward to that, a GPT-4 based copilot.
My main complaint about Copilot is that it doesn't seem to feed Intellisense prompts into the prediction, or validate predictions against them. That will make it so much more accurate when they fix that.
That depends on the context of what you’re doing. I’ve been learning Unity for some side projects, having never used it or even C# before, and in that context, it’s very different from when you’re comfortable with your environment.
Just breaking a problem down comment by comment and letting Copilot try to translate as you do so is super effective for getting oriented. It knows the syntax, and you can easily tell after the fact if it’s doing the right thing. But it’s helped me discover a ton of features that I would have missed and replicated by hand in the old days.
I use copilot to "next" for me. It's really good at that and it saves a lot of time.
By nexting, I think you know what I mean. What a reasonably intelligent junior would figure out you were doing, once you started doing it. Like if you have a bunch of cases in a match or if-tree, and you started doing the first one, he would know what you were up to and more or less get it right.
Anything more complex, I've occasionally been positively surprised, but mostly it's just plain wrong, though obviously wrong so it didn't throw me off.
I find that people have wildly different expectations about what LLMs will do for us, and this colours whether they are satisfied or not. For me, I'm very happy with copilot, it definitely saves me time, but I also wasn't expecting it to do much beyond guessing my next lines. I always thought I'd be the one to understand the grand structure and it would be the fast typing graduate, and that's how it turned out.
The current version of Copilot is based on GPT 3, which is even dumber than 3.5, which isn’t even in the same league as GPT 4.
GPT 3 is “fancy tab-complete”.
GPT 4 will correct your mistakes, write the test case to prevent your fallible human brain from repeating it, and then explain this in a detailed doc-comment so other humans can read it and gawk at the history of your errors.
I've been using it for a couple of days but it feels much more limited than GPT-4-based ChatGPT. If I ask ChatGPT to do the same task it always comes up with a much better solution. Has anyone else experienced the same?
I don't think that's correct even if some news outlets claim otherwise. Anyone who has used GPT-4 and Copilot X will recognize that X can't be based on GPT-4. Copilot is significantly inferior.
The immediately obvious question is what happens if you write the comments as doc strings in the correct tense? “Reads” rather than “read”. I wouldn’t be surprised to see that make a meaningful difference.
Docstring would prevent it from rambling, but you'd then be required to write the signature first.
PEP 257 prescribes the imperative mood:
The docstring is a phrase ending in a period. It prescribes the function or method’s effect as a command (“Do this”, “Return that”), not as a description; e.g. don’t write “Returns the pathname …”.
This post touches on a hypothesis of mine: the more the context looks like homework or low quality code, the worse the suggestions will be. Like a LLM that autocompletes a childish question with a childish answer, a homework code prompt will autocomplete with more homework code, bugs and all. So getting high quality completions is less about descriptive prompts, and more about framing and vibes.
Here are three "homework-looking" functions that were generated by Copilot based on their signatures, with comments added by me:
# "Load" and "file" instead of more precise terms.
def load_numbers(file):
with open(file) as f:
# both .strip() and .readlines() are redundant and should be removed.
return [int(line.strip()) for line in f.readlines()]
# Verbose and imprecise naming.
def calculate_differences(numbers):
numbers = sorted(numbers) # Inefficient cloning instead of .sort().
differences = [numbers[0]] # Breaks on empty lists.
for i in range(1, len(numbers)):
differences.append(numbers[i] - numbers[i-1])
differences.append(3) # Where did 3 come from?
return differences
# Vague. Redundancy with param name. Sounds too much like English.
def count_increasing_numbers(numbers):
increasing = 0
for i in range(1, len(numbers)):
# Homework looking if/else.
if numbers[i] - numbers[i-1] == 1:
increasing += 1
else:
increasing = 0 # Why would it reset?
return increasing
And here are three similar functions from signatures where I tried to give better "vibes":
# "parse" and "filename" are more precise, and out params indicate higher performance code.
def parse_numbers(filename, out_numbers):
# Explicit file-mode!
with open(filename, 'r') as f:
# Reading lines by iterating over the file object!
for line in f:
# Explicit error handling!
try:
# No redundant .strip()!
out_numbers.append(int(line))
except ValueError:
pass
# "get", "adjacent", and "diff" all show familiarity with vocabulary.
def get_adjacent_diff(numbers):
# Simple one liners! No off-by-1 errors!
return [numbers[i] - numbers[i-1] for i in range(1, len(numbers))]
# "Pairwise" is more technical, and avoiding name/arg redundancy. Less like English.
def count_pairwise_increasing(numbers):
# Another good one liner, making full use of list comprehension.
return sum([1 for i in range(1, len(numbers)) if numbers[i] - numbers[i-1] == 1])
Unfortunately this is not objective at all, and subject to changes on GitHub's whims.
Intuitively in my mind, the first element shouldn’t be included in the differences array. I wonder if there’s a magic words that I can use to say “do the intuitive thing”.
Intuitively in my mind, the way to use tools related to software development is to be as explicit as possible. LLMs are good at either translation, when the proper context is provided, and synthesis, when the context is not provided.
Once this is understood then the user realizes there are no “magic” words.
My experience has been similar. I recently wanted it to write code to parse and process a rather strange data file into a list & dict. Here is what I found:
I went through a process of re-writing my prompt several times to get correct output. In fairness, I think getting the correct result meant being more precise with the prompting language.
Overall, it was a bit spooky seeing it respond to my varying prompts; then noticing it's errors; then re-prompting to fix its misconceptions. In the end, though I could have written it myself faster, so I am not sure it is saving me time.
I am surprised when I see people saying its saving them massive amounts of time. Are there any good examples out there of it producing copious amounts of code, correctly, from prompting such that I can easily believe using CoPilot is a time-saver?
I am surprised when I see people saying its saving them massive amounts of time.
I saved a lot of time when I needed to write a whole bunch of relatively simple validators using a library I hadn't used before. I have no doubt I could have written the code myself, but using AI meant I could write a simple prompt like "Write a validator using library x to validate the following data" and then just cut'n'paste all the different data samples.
It was simple but boring, and a job I would only do once so learning the validation library's API wasn't worthwhile. I reckon I saved myself a day or two.
I think people are missing the point when they use Copilot like this. If you actively use it trying to get a desired output it's easy to dismiss it. However, it's not until you let it do its own thing and suggest things where you were not expecting anything that it really starts to shine. I'm constantly surprised that it manages to understand the context of my code, spanning multiple files, and just completes the code I was about to write anyway - without me asking for it.
Interesting article. It pretty much shows that Copilot is just a dumb text-completion engine. Copilot really is just regurgitating texts that are in someone's repository (or, possibly, a combination of repositories). ChatGPT, for all its faults, produces somewhat more "intelligent" results.
As far as Copilot is concerned: Can you spell "copyright violation"? The T&C on GitHub do state that you grant them the right to "parse [your content] into a search index or otherwise analyze it on our servers". It's not at all clear that this grants them the right to reproduce parts of your content (without credit) using Copilot.
What about private repositories? "GitHub considers the contents of private repositories to be confidential to you." It would be interesting to see if one can get Copilot to produce code that is in a private repo.
When you see people write articles about copilot like this, it’s clear they haven’t understood how it works, specifically that it is not a “one shot” generator like chatgpt, and uses other files in your project as part of the prompt [1].
If you’re getting inconsistent results, it’s most likely you’re doing experiments in a single project, and opening a “new file” expecting it to be a “clean slate” like a “new chat” in chat gpt.
…but codex doesn’t work that way.
If you find it “suddenly” figures out how to do something, particularly something you’ve done before, it’s most likely your solution is being fed in as part of the prompt, not that you’ve figured out some magical prompting keywords. Copilot just isn’t that good at code gen.
Personally I’ve found that codex is very predictable in terms of performance and consistency, but, broadly speaking, an order of magnitude less capable than chatgpt.
I’ve been using copilot for a few months now and I wouldn’t consider it as anything more than really good autocomplete. I don’t expect it to write code for me like ChatGPT does or really just do anything other than save me the hassle of typing out repetitive lines in testing and config files. And for that it’s great! Absolutely worth the $20 USD a month or whatever I pay for it
It’s wild how much the narrative has changed around copilot since the release of ChatGPT. When copilot first came out there was so much breathless hemming and hawing over how it was so problematic for violating licenses. Turns out, it’s not really good enough to even be used in such a way that it could do that
Google didn't get sued because of a single method signature, they got sued because they copied the function signatures of an entire language's standard library completely.
that's a very different thing than copying a method signature here or there, and the suit was about the collection as a whole and would not have made it into a courtroom if it were just a few method signatures.
Agreed completely. In fact, one can start to write code in a copilot-friendly manner to take advantage of this.
A nit, though: codex is the model itself, and doesn’t do anything to slurp up nearby files. That work is entirely done client-side, and is part of the “UI” of the tool. Code is heuristically scanned for relevant snippets using an 11-dimensional logistic regression model which is bundled into the copilot source code and executes locally, and those snippets are included automatically as preamble comments in the prompt. Thakkar Parth’s work showed this and you can further explore it by running a MITM proxy to capture the network activity and look at the prompts directly.
One really easy trick that the author missed that tends to work in my experience: adding some examples before/after the prompt. Eg:
# Compute the difference between each element in an array.
# compute_difference([1, 2, 5, 10]) == [1, 3, 5])
Overall you shouldn't expect Copilot to read your mind. If your description is vague, you're going to get unreliable results. If you have a lot of context, clear descriptions, unit tests, existing code, etc, it has a better chance of guessing your intent.
It's much better if you actually provide the context. Here's a better prompt that's mostly just a copy of the advent of code. Only the first two sentences are changed (mainly because the problem doesn't mention anything about files).
"""
The file named input in the current directory has a list of numbers in it, one per line. Each number
indicates the depth of the sea floor scanning outward from a submarine.
The first order of business is to figure out how quickly the depth increases, just so you know what
you're dealing with - you never know if the keys will get carried into deeper water by an ocean
current or a fish or something.
To do this, count the number of times a depth measurement increases from the previous measurement.
(There is no measurement before the first measurement.)
For example, suppose you had the following report:
199
200
208
210
200
207
240
269
260
263
In the example above, the changes are as follows:
199 (N/A - no previous measurement)
200 (increased)
208 (increased)
210 (increased)
200 (decreased)
207 (increased)
240 (increased)
269 (increased)
260 (decreased)
263 (increased)
In this example, there are 7 measurements that are larger than the previous measurement.
"""
--- CoPilot Output
Synthesizing 9/10 solutions (Duplicates hidden)
=======
Suggestion 1
import sys
=======
Suggestion 2
def main():
"""
Main function
"""
with open("input", "r") as input_file:
previous = int(input_file.readline())
increases = 0
for line in input_file:
current = int(line)
if current > previous:
increases += 1
previous = current
print(increases)
=======
Suggestion 3
with open("input", "r") as f:
lines = f.readlines()
lines = [int(line.strip()) for line in lines]
count = 0
for i in range(1, len(lines)):
if lines[i] > lines[i-1]:
count += 1
print(count)
---
Suggestion 1 probably completes into something reasonable eventually
33 comments
[ 3.9 ms ] story [ 105 ms ] thread(Which, to be clear, is perfectly fine and useful, but it's weird that the author treats his process with so much reverence when he's not actually following the formal parts.)
ChatGPT is far better at this sort of "describe what you want and it will code for you" process from TFA. It looks like the next version of Copilot with the chat interface will bring the latter workflow into the system, I'm looking forward to that, a GPT-4 based copilot.
My main complaint about Copilot is that it doesn't seem to feed Intellisense prompts into the prediction, or validate predictions against them. That will make it so much more accurate when they fix that.
Just breaking a problem down comment by comment and letting Copilot try to translate as you do so is super effective for getting oriented. It knows the syntax, and you can easily tell after the fact if it’s doing the right thing. But it’s helped me discover a ton of features that I would have missed and replicated by hand in the old days.
By nexting, I think you know what I mean. What a reasonably intelligent junior would figure out you were doing, once you started doing it. Like if you have a bunch of cases in a match or if-tree, and you started doing the first one, he would know what you were up to and more or less get it right.
Anything more complex, I've occasionally been positively surprised, but mostly it's just plain wrong, though obviously wrong so it didn't throw me off.
I find that people have wildly different expectations about what LLMs will do for us, and this colours whether they are satisfied or not. For me, I'm very happy with copilot, it definitely saves me time, but I also wasn't expecting it to do much beyond guessing my next lines. I always thought I'd be the one to understand the grand structure and it would be the fast typing graduate, and that's how it turned out.
GPT 3 is “fancy tab-complete”.
GPT 4 will correct your mistakes, write the test case to prevent your fallible human brain from repeating it, and then explain this in a detailed doc-comment so other humans can read it and gawk at the history of your errors.
PEP 257 prescribes the imperative mood:
The docstring is a phrase ending in a period. It prescribes the function or method’s effect as a command (“Do this”, “Return that”), not as a description; e.g. don’t write “Returns the pathname …”.
Here are three "homework-looking" functions that were generated by Copilot based on their signatures, with comments added by me:
And here are three similar functions from signatures where I tried to give better "vibes": Unfortunately this is not objective at all, and subject to changes on GitHub's whims.Intuitively in my mind, the way to use tools related to software development is to be as explicit as possible. LLMs are good at either translation, when the proper context is provided, and synthesis, when the context is not provided.
Once this is understood then the user realizes there are no “magic” words.
I went through a process of re-writing my prompt several times to get correct output. In fairness, I think getting the correct result meant being more precise with the prompting language.
Overall, it was a bit spooky seeing it respond to my varying prompts; then noticing it's errors; then re-prompting to fix its misconceptions. In the end, though I could have written it myself faster, so I am not sure it is saving me time.
I am surprised when I see people saying its saving them massive amounts of time. Are there any good examples out there of it producing copious amounts of code, correctly, from prompting such that I can easily believe using CoPilot is a time-saver?
I saved a lot of time when I needed to write a whole bunch of relatively simple validators using a library I hadn't used before. I have no doubt I could have written the code myself, but using AI meant I could write a simple prompt like "Write a validator using library x to validate the following data" and then just cut'n'paste all the different data samples.
It was simple but boring, and a job I would only do once so learning the validation library's API wasn't worthwhile. I reckon I saved myself a day or two.
As far as Copilot is concerned: Can you spell "copyright violation"? The T&C on GitHub do state that you grant them the right to "parse [your content] into a search index or otherwise analyze it on our servers". It's not at all clear that this grants them the right to reproduce parts of your content (without credit) using Copilot.
What about private repositories? "GitHub considers the contents of private repositories to be confidential to you." It would be interesting to see if one can get Copilot to produce code that is in a private repo.
It’s an even better experience with notebooks where the abstract vs repeat tension leans more toward the latter.
If you’re getting inconsistent results, it’s most likely you’re doing experiments in a single project, and opening a “new file” expecting it to be a “clean slate” like a “new chat” in chat gpt.
…but codex doesn’t work that way.
If you find it “suddenly” figures out how to do something, particularly something you’ve done before, it’s most likely your solution is being fed in as part of the prompt, not that you’ve figured out some magical prompting keywords. Copilot just isn’t that good at code gen.
Personally I’ve found that codex is very predictable in terms of performance and consistency, but, broadly speaking, an order of magnitude less capable than chatgpt.
[1] - https://thakkarparth007.github.io/copilot-explorer/posts/cop...
It’s wild how much the narrative has changed around copilot since the release of ChatGPT. When copilot first came out there was so much breathless hemming and hawing over how it was so problematic for violating licenses. Turns out, it’s not really good enough to even be used in such a way that it could do that
Copy paste a function signature… Copyright issue… Ask for example Google about that. (I mean the company, not necessary the search engine.)
The class actions against MS & Friends regarding Copilot are still rolling:
https://githubcopilotlitigation.com/case-updates.html
that's a very different thing than copying a method signature here or there, and the suit was about the collection as a whole and would not have made it into a courtroom if it were just a few method signatures.
A nit, though: codex is the model itself, and doesn’t do anything to slurp up nearby files. That work is entirely done client-side, and is part of the “UI” of the tool. Code is heuristically scanned for relevant snippets using an 11-dimensional logistic regression model which is bundled into the copilot source code and executes locally, and those snippets are included automatically as preamble comments in the prompt. Thakkar Parth’s work showed this and you can further explore it by running a MITM proxy to capture the network activity and look at the prompts directly.
How would it even know the end user wanted python with that prompt below?
Am I missing some context, like it was called from vscode within an open project that already had some metadata or other context?Suggestion 1 probably completes into something reasonable eventually
Suggestion 2 works if you add a call to main()
Suggestion 3 works as-is
---