You could train one to recognize common patterns, and you could train one to heuristically guess that an algo running on test inputs is looping forever if it takes too long, but actually dividing the world into "loops forever" and "does not" is the Halting Problem, and our solution is guaranteed to either false-positive or false-negative.
(Anecdote: I heard a story once about a college that transitioned from free printing to metered printing, but since the printers didn't support metering, they implemented it by running print jobs in a virtual machine and tallying page count there. It took not very much time for some compsci student to cobble together a script that would augment a Postscript file with an "if my environment is virtual, print one blank page and exit, else..." prefix).
As you point out elsewhere, there's a human in the loop. Though the Halting Problem applies to human minds as well, they likely won't have any difficulty identifying useless code the ML doesn't pick up and adding it to the training set.
The next step is to integrate actual parts of the software into the poison-pill, so that if the poison-pill is not included then the software doesn't work.
It isn't necessarily a good idea, but it's the next obvious step.
That's true if the same mitigations often show up with the same contexts, like the simplistic one in the example. Making a general mitigation-avoiding AI would require it to understand the intention of the code.
I've never learned Haskell, so I'm maybe misunderstanding this, but it seems like this wouldn't cause problems for Copilot.
My understanding is that the author is effectively booby-trapping the entrypoints into their applications so that if someone runs the same code without the right "antidote" code extensions, it will go into an infinite loop. But that's only effective if Copilot picks up that one line with the booby trap. And Copilot probably wouldn't ever suggest that line because it's so different than the entrypoint to most Haskell applications.
I wouldn't underestimate the coming arms race though, as developers find newer and more subtle poison pills to prevent rogue AIs from ingesting their code and regurgitating it later.
If someone is poisoning and obfuscating their own code base, they become the only person who can contribute to it, defeating the point of being FOSS and benefiting from community contributions and review.
This is being downvoted, but it touches on a really interesting phenomena we may eventually see, a sort of "arms race" where people try to make their code useless to machine learning, but simultaneously make it useless to people with lower skills (people blindly copying/pasting code with little understanding types).
As the machine learning becomes "smarter", this bar will raise, and more people will be coincidentally excluded in an attempt to keep unwanted algorithms from sucking up code.
Assuming this sort of poison-pill thing ever becomes commonplace.
The humans have been blindly copy-pasting from Stackoverflow for years, this simplifies that process. Especially for 10X remote engineers Copilot is a godsend.
For that result to occur, you would need so much poisoned code that it became the dominant piece of the training data.
At that point, open source code becomes hard for humans to reason about. Very much cutting off one's nose despite one's face for the open source community.
Don't worry; Roko's Basilisk has no incentive to torture future-you. You can't predict it well enough to know whether it will, so it having the general policy of “follow through on the torture” won't help it – and will, in fact, just waste resources. (Unless it inherently cares about torturing people, but nobody has any incentive to build such a thing – and you should be scared enough of whoever's trying even if they don't succeed.)
Roko's Basilisk only works if you're a superintelligence willing to risk being counterfactually blackmailed in this way. Which you're not. You're safe.
Just hoping it will keep me around as a pet, or that either the governator or Emilia Clarke come barging in telling me to come with them if I want to live.
I think Github should support some kind of variant of robots.txt in the source code. Maybe a common line in the license file that excludes it from ML training data. That said, if you want to exclude it, that's your right, but you should consider why you are open sourcing in the first place, and what about ML feels different to you than traditional human cognition.
Why should the users be forced to do extra work in order for Github to do their commercialization "correctly"?
This is something Github/Microsoft started, something that only really benefits Github/Microsoft and that they were doing lackadaisical from the start by ignoring licenses.
Github ignored licenses for this project and included code from all publicly-accessible repos. They claim it's not copyright infringement and you can e.g. use code from this AI trained on open-source repos in closed-source projects.
It's not clear to me how it is copyright infringement if the generated code is truly new. Of course, if some code output is very similar to existing code, then it is obviously copyrighted.
When you read code from an open source GPL repo and are inspired by it to write your own code, it's obviously not copyrighted.
When I read code, I understand it, its intentions, and its output. I'm not simply replicating things that I've read before. I learned about inheritance in java in school. When I go to write python, I need only look up the syntax because I understand the logic.
One problem is that copyright is not well-defined. They just have to convince a court that their use is "transformational" and not a "derivative work" and they're home free. Then it doesn't matter what the license says, because Microsoft isn't licensing your work.
No. Opt-out is exactly the wrong approach. Instead consider a flag indicating that "I'm OK with mu source being used for training AI models." i.e. opt-in.
The problem is that defaults are very powerful. I could equally well argue that the Internet Archive shouldn't archive my website unless I give them opt-in permission. Of course, if that were the case, the Internet Archive effectively wouldn't exist.
ADDED: To be clear, it may be fine to say that making certain things opt-in means that other things that many might value can't exist.
The counterpoint here is that the internet archive isn’t a for profit service which is claiming exact reproductions of other people’s content as “new work”
Things need to be made opt-in because we want there to be a meaningful barrier to companies using data that does not belong to them in order to make money.
Microsoft GitHub Copilot regardless of legality to me is a direct attack on OSS in an EEE manner. It is an abuse of the platform. It should have been opt-in. It pushed me ahead to remove my repos from GitHub over to Gitlab (love that instant import feature), repos will be private by default. I will be deleting my Microsoft accounts too. Already off Windows.
I also think the software development community should consider poisoning the well, even if it’s already legally poisoned.
People have sued over similar stuff in the past, like Google training their language model on copyrighted materials. The legal consensus is that a model trained on a large volume of copyrighted works does not infringe copyright, even if it occasionally vomits up chunks of the input model.
That said, if you trained a model specifically designed to violate copyright I suspect the courts would have a different opinion. They consider intent just as much as (if not more than) what you actually did.
This might sound stupid, and maybe it is, but that’s the state of law right now. It’s possible somebody will drag out a big slam dunk case that overturns much of this or that new laws will be passed… but right now (as far as I can tell) Copilot looks pretty legal.
(Of course, talk to your legal counsel, this may vary be region, etc.)
Assuming GitHub isn’t lying, no it isn’t. They claim the vast majority of output is original, much like how GPT3 typically generates original text (although can sometimes output inputs verbatim). If that’s their intent and they haven’t deliberately tried to make their model output it’s inputs verbatim, then they’re good. Remember, there’s a big sampling bias online towards people sharing bits of code that are obvious verbatim inputs - you don’t see all the ones that aren’t.
What you say only makes sense if your code is licensed under something similar to public domain. Some licenses, like the GPL, require derivative works to use the same license.
No, simply reading GPL code doesn't poison you (though it does put you in an awkward position), but copy pasting gpl code straight into your codebase is a copyright violation unless you can prove fair use. And copy pasting GPL code is what Microsoft Copilot has been repeatedly shown to do.
First of all, none of this matters if the resulting code isn't distributed in some manner. For an internal app, for example, none of this matters.
Secondly, it's pretty much obvious that fairly trivial (and, frankly, probably not so trivial) code snippets are reused/reworked from various sources all the time. This makes it hard for me to see Copilot as this new and existentially harmful thing.
That's true, but some of the copy pasting that Copilot indulges in would absolutely fulfill that requirement. See: copilot copy pasting quake's inverse square root function: https://twitter.com/mitsuhiko/status/1410886329924194309
Copy pasting a one-line snippet may be OK, but copy pasting a full function is already much more of a grey area.
This fast inverse square root function is also getting old. If it was a software patent, which are not recognized in many places, it would have expired a few years ago.
From what I can see it has only been "repeatedly shown to do" it when you explicitly try to force it to do it (e.g. adding those Quake specific comments to bring out the fast inverse square root code). I want to see a serious analysis of Microsoft's claim that it only happens 0.1% of the times. Has it actually been debunked?
It is only an attack on OSS if licenses like the GPL are not respected. If copilot simply copies code it was trained on, it is more of a threat to proprietary software, since any sufficiently large copy from GPL'd software would have to be GPL licensed too.
If GPL is respected, copilot may help many proprietary softwares to be open source re-licensed. That should be seen as a positive outcome from FLOSS supporters.
But it is not easy to predict exactly what will happen and how people and companies will behave.
It seems like it would be trivial to poison the well so to speak for common function prototypes to add known vulnerabilities that can be exploited later when needed.
I would be exceedingly cautious about running code inserted by a black box fed by public inputs. Similar attacks have already occurred for NPM and I'm sure many other packagers.
I find the Copilot and AI discussions quite interesting.
On the one side its indeed problematic that Microsoft (and others in the industry in general) hardly ever ask for permission for data to train models.
On the other side, if you consider AI and computers to be similar to humans, then it's not an issue at all because humans also learn by reading code or information from various sources that are copyrighted or stuff they havent paid for directly (eg a library or "I'm just browsing in a book store" are an interesting for example, the later some books stores actually dont like).
For poisoning the well kind of tactics, the author forgets that you poison the well for everyone - not just ML.
An issue not brought into the mix with Copilot is that computers are superior in their memory and the amount of learning they can do (even now already in specialized domains), and hence maybe the fear, long term issues (incl ethical ones) and people being upset should actually come from that angle.
You know, a computer that can reprogram itself seems like maybe a bad idea.
Yeah I have seen a lot of people bring up the question of essentially how training an AI to regurgitate code snippets is different from a human reading through and learning how to write similar code.
To me they seem obviously and intrinsically different, and I think your post gets at some of the reasons. For one, an AI can ingest a much larger quantity of data much faster than a human. No human could possibly read all of the public code on GitHub in one lifetime. But more importantly, the method of ingestion and synthesis, as well as the nature of the application of what’s learned, is vastly different. Generally a human reading through OSS code to learn isn’t going to then permanently store verbatim segments of it in their memory to regurgitate later. They may copy a bit of code while learning, but that then morphs into an independent understanding of the type of problem being solved and one way to approach that problem. Copilot OTOH has no capacity to understand different approaches to solving a problem, to compare one against the other, and so on. All it knows is which is used more frequently, with no capacity to map that against metrics of long term maintenance, CI suite success, readability, and so on.
As for the poison pill problem, I think it actually illustrates the difference quite well. A simple comment explaining that the code is a booby trap would be enough for any human reading the code to know how to avoid it. The AI on the other hand has no capacity to understand that comment and to know not to put the code into its dataset.
> A simple comment explaining that the code is a booby trap would be enough for any human reading the code to know how to avoid it. The AI on the other hand has no capacity to understand that comment and to know not to put the code into its dataset.
Copilot users can be leveraged to train the algorithm to emit less such comments/traps. Copilot inserts snippets by having an extension running in the editor, watching what is typed and responding. That's an opportunity to gather more training labels. For example, which lines of Copilot emitted code are frequently deleted soon after being inserted? Gather enough such training labels and the algorithm can start to recognize many of those comments (and/or the traps the comments call out) and stop emitting those specific lines.
At that point, warning comments help the algorithm better distinguish desirable lines of code from traps. And you'll be less likely to have traps leak into your code by using Copilot rather than copy/pasting snippets from the web (both are subject to human review, but Copilot will already have filtered some traps). So it's possible that widespread deployment of traps would drive adoption of Copilot by people who frequently copy/paste code from the web.
If you consider AI to be similar enough to humans to directly map our ideas about copyright and fair use, then copyright is way down the list of concerns we need to address. First we would need to arrange wages for each ML system's labor, as well as considering how to handle voting rights.
Keep in mind that CoPilot might itself be a well-poisoning tactic.
If use of Github presumes use of CoPilot and of comingled code under incompatible, or proprietary, licenses, such that use of such code could then create contributory infringement claims against distributors, users, or developers, there's something of a problem across the Free Software world.
Though that does seem rather a bit of a major footgun for Github / Microsoft themselves.
The Free Software movement didn't create, or even want, a world in which copyrighted software was a norm. But it adpated to the circumstance by treating copyright as a serious matter and being diligent in practices of use, appropriation, and licencing.
It's rather ironic that the source of the "Letter to Hobbyists"[1] are now advocating a devil-may-care attitude to copyright, software, and licensing in their own works and service offerings.
It's time build an open source free co-pilot. When people are actively contributing to an AI system, my guess is that it will fare good against rot scanned ones.
[programmers, for years and years] Hi we built a huge industry around pointing AIs at normies’ comms and renting it out to advertisers etc. This is very legal and you are Bad if you try to fight it copyright grounds. Indexing and downloading stuff is a basic human right and always Fair Use!
[programmers when you do the exact same thing to their publicly shared code] How dare you, this is grounds for vigilante justice!11!1
1. This is worse than cutting off your nose to spite your face. I can easily imagine some config being messed up, and then the author shoots themselves in the foot. Also, god forbid anyone else have maintain the Rube Goldberg machine you've decided to create.
2.The idea that this could provide some significant dent to Copilot's utility is laughable. First off, it's not hard to see how AI could catch stuff like this. But more importantly, Copilot is clearly and obviously not intended to just be paste and apply. Every comment I've seen from alpha testers has stated how Copilot is uncannily good, but all the code must still be reviewed and it is oftentimes just flat out off. Code like this that would fail immediately would easily be identified.
I understand the arguments over Copilot, but this is just a dumb response to it.
At the end of the day source code is for human readability. If you pollute your code with a poison pill you just make it harder for other humans to use.
ML Systems are doing sequence analysis and looking to predict the next token, for an arbitrary length of phrase. You've poisoned your code but compared to the volume of GitHub or any other public code data set it becomes the equivalent of noise. If you create a movement that puts poison pills on all Haskell files, now every project has painful boilerplate and an AI will learn to produce the same code. And every human that deals with it won't see a difference because you created a Haskell standard.
It's interesting to compare Copilot with Wingman for Haskell: https://youtu.be/S0HvfXq3454?t=550 Both attempts at somewhat automating repetitive aspects of programming, but with completely different approaches. Wingman was inspired by tactic metaprogramming in theorem provers like Coq.
97 comments
[ 3.2 ms ] story [ 192 ms ] thread(Don't shoot the messenger; the concept is obvious.)
(Anecdote: I heard a story once about a college that transitioned from free printing to metered printing, but since the printers didn't support metering, they implemented it by running print jobs in a virtual machine and tallying page count there. It took not very much time for some compsci student to cobble together a script that would augment a Postscript file with an "if my environment is virtual, print one blank page and exit, else..." prefix).
It isn't necessarily a good idea, but it's the next obvious step.
My understanding is that the author is effectively booby-trapping the entrypoints into their applications so that if someone runs the same code without the right "antidote" code extensions, it will go into an infinite loop. But that's only effective if Copilot picks up that one line with the booby trap. And Copilot probably wouldn't ever suggest that line because it's so different than the entrypoint to most Haskell applications.
As the machine learning becomes "smarter", this bar will raise, and more people will be coincidentally excluded in an attempt to keep unwanted algorithms from sucking up code.
Assuming this sort of poison-pill thing ever becomes commonplace.
At that point, open source code becomes hard for humans to reason about. Very much cutting off one's nose despite one's face for the open source community.
Hold my beer...
Not just. The linked example is booby-trapping a function.
Also, I'd like to propose that henceforth, we refer to GitHub's mascot "Octocat" as "Copycat". All those in favour...
Roko's Basilisk only works if you're a superintelligence willing to risk being counterfactually blackmailed in this way. Which you're not. You're safe.
Just hoping it will keep me around as a pet, or that either the governator or Emilia Clarke come barging in telling me to come with them if I want to live.
Roko's Basilisk is no joke.
I like the new mascot.
People need to stop referring to the product name Github, too, as it causes confusion. It's _Microsoft_, which should serve as a deterrent.
This is something Github/Microsoft started, something that only really benefits Github/Microsoft and that they were doing lackadaisical from the start by ignoring licenses.
It's not clear to me how it is copyright infringement if the generated code is truly new. Of course, if some code output is very similar to existing code, then it is obviously copyrighted.
When you read code from an open source GPL repo and are inspired by it to write your own code, it's obviously not copyrighted.
Also, some people explicitly avoid reading codebases to avoid duplicating by mistake some copyrighted code.
Not sure if this is true but I heard that it used to be forbidden to Microsoft developers to read open source code.
ADDED: To be clear, it may be fine to say that making certain things opt-in means that other things that many might value can't exist.
This isn't the problem, it's the whole point.
Things need to be made opt-in because we want there to be a meaningful barrier to companies using data that does not belong to them in order to make money.
I also think the software development community should consider poisoning the well, even if it’s already legally poisoned.
I get that it still makes a statement.
Someone should also release a model trained on the leaked Windows source code ..
That said, if you trained a model specifically designed to violate copyright I suspect the courts would have a different opinion. They consider intent just as much as (if not more than) what you actually did.
This might sound stupid, and maybe it is, but that’s the state of law right now. It’s possible somebody will drag out a big slam dunk case that overturns much of this or that new laws will be passed… but right now (as far as I can tell) Copilot looks pretty legal.
(Of course, talk to your legal counsel, this may vary be region, etc.)
Secondly, it's pretty much obvious that fairly trivial (and, frankly, probably not so trivial) code snippets are reused/reworked from various sources all the time. This makes it hard for me to see Copilot as this new and existentially harmful thing.
It is harmful to open-source, because if these wishes are not respected, people will stop contributing and stop open-sourcing things.
This is not true. The copied code needs to be distinct and uncommon; a tiny snippet is very unlikely to meet this requirement.
Copy pasting a one-line snippet may be OK, but copy pasting a full function is already much more of a grey area.
In the US, the copyright lapses 95 years from publication, or 120 years from creation, or 70 years from author death, whichever is shortest.
So the inverse square root function will stay copyrighted until 2100 (the source was published in 2005 I believe).
No one should use this, because you could easily be a derivative of something else and you are not in control of your own license.
If you copy something from a BSD licensed software you should credit them.
If GPL is respected, copilot may help many proprietary softwares to be open source re-licensed. That should be seen as a positive outcome from FLOSS supporters.
But it is not easy to predict exactly what will happen and how people and companies will behave.
Information want to be free.
I would be exceedingly cautious about running code inserted by a black box fed by public inputs. Similar attacks have already occurred for NPM and I'm sure many other packagers.
On the one side its indeed problematic that Microsoft (and others in the industry in general) hardly ever ask for permission for data to train models.
On the other side, if you consider AI and computers to be similar to humans, then it's not an issue at all because humans also learn by reading code or information from various sources that are copyrighted or stuff they havent paid for directly (eg a library or "I'm just browsing in a book store" are an interesting for example, the later some books stores actually dont like).
For poisoning the well kind of tactics, the author forgets that you poison the well for everyone - not just ML.
An issue not brought into the mix with Copilot is that computers are superior in their memory and the amount of learning they can do (even now already in specialized domains), and hence maybe the fear, long term issues (incl ethical ones) and people being upset should actually come from that angle.
You know, a computer that can reprogram itself seems like maybe a bad idea.
To me they seem obviously and intrinsically different, and I think your post gets at some of the reasons. For one, an AI can ingest a much larger quantity of data much faster than a human. No human could possibly read all of the public code on GitHub in one lifetime. But more importantly, the method of ingestion and synthesis, as well as the nature of the application of what’s learned, is vastly different. Generally a human reading through OSS code to learn isn’t going to then permanently store verbatim segments of it in their memory to regurgitate later. They may copy a bit of code while learning, but that then morphs into an independent understanding of the type of problem being solved and one way to approach that problem. Copilot OTOH has no capacity to understand different approaches to solving a problem, to compare one against the other, and so on. All it knows is which is used more frequently, with no capacity to map that against metrics of long term maintenance, CI suite success, readability, and so on.
As for the poison pill problem, I think it actually illustrates the difference quite well. A simple comment explaining that the code is a booby trap would be enough for any human reading the code to know how to avoid it. The AI on the other hand has no capacity to understand that comment and to know not to put the code into its dataset.
Copilot users can be leveraged to train the algorithm to emit less such comments/traps. Copilot inserts snippets by having an extension running in the editor, watching what is typed and responding. That's an opportunity to gather more training labels. For example, which lines of Copilot emitted code are frequently deleted soon after being inserted? Gather enough such training labels and the algorithm can start to recognize many of those comments (and/or the traps the comments call out) and stop emitting those specific lines.
At that point, warning comments help the algorithm better distinguish desirable lines of code from traps. And you'll be less likely to have traps leak into your code by using Copilot rather than copy/pasting snippets from the web (both are subject to human review, but Copilot will already have filtered some traps). So it's possible that widespread deployment of traps would drive adoption of Copilot by people who frequently copy/paste code from the web.
If use of Github presumes use of CoPilot and of comingled code under incompatible, or proprietary, licenses, such that use of such code could then create contributory infringement claims against distributors, users, or developers, there's something of a problem across the Free Software world.
Though that does seem rather a bit of a major footgun for Github / Microsoft themselves.
The Free Software movement didn't create, or even want, a world in which copyrighted software was a norm. But it adpated to the circumstance by treating copyright as a serious matter and being diligent in practices of use, appropriation, and licencing.
It's rather ironic that the source of the "Letter to Hobbyists"[1] are now advocating a devil-may-care attitude to copyright, software, and licensing in their own works and service offerings.
________________________________
Notes:
1. See: https://genius.com/Bill-gates-an-open-letter-to-hobbyists-an...
[programmers when you do the exact same thing to their publicly shared code] How dare you, this is grounds for vigilante justice!11!1
But, all’s fair in love and war. If I publish some code and your AI bot thing happens to choke on it.... write a better bot, don’t blame me.
1. This is worse than cutting off your nose to spite your face. I can easily imagine some config being messed up, and then the author shoots themselves in the foot. Also, god forbid anyone else have maintain the Rube Goldberg machine you've decided to create.
2.The idea that this could provide some significant dent to Copilot's utility is laughable. First off, it's not hard to see how AI could catch stuff like this. But more importantly, Copilot is clearly and obviously not intended to just be paste and apply. Every comment I've seen from alpha testers has stated how Copilot is uncannily good, but all the code must still be reviewed and it is oftentimes just flat out off. Code like this that would fail immediately would easily be identified.
I understand the arguments over Copilot, but this is just a dumb response to it.
At the end of the day source code is for human readability. If you pollute your code with a poison pill you just make it harder for other humans to use.
ML Systems are doing sequence analysis and looking to predict the next token, for an arbitrary length of phrase. You've poisoned your code but compared to the volume of GitHub or any other public code data set it becomes the equivalent of noise. If you create a movement that puts poison pills on all Haskell files, now every project has painful boilerplate and an AI will learn to produce the same code. And every human that deals with it won't see a difference because you created a Haskell standard.
This could be seen as a form of protest or as a Luddite (https://en.wikipedia.org/wiki/Luddite) reaction to AI automation of coding.
James Micken's warning against connecting machine learning to the Internet of Hate still apply: https://youtu.be/ajGX7odA87k?t=1206
I understand that it may feel good to be mad at large corporations but GitHub Inc. is still independent from MS.