The article points out that training data generated using ChatGPT is necessarily biased or tainted with the consequences of the policy optimizations and RLHF alignment processes conducted by OpenAI. This results in models that reflect the alignment preferences of OpenAI instead of the preferences of the model developers.
Indeed, & an artificial uniformity of LLMs, and thought, if everyone is cribbing each others' outputs could be a concern.
But before its concern about a monoculture, the article 1st points out that mere prediction-training (on another model's outputs or fresh data) can't truly match RLHF in instilling some much-desired behaviors.
And that presents a bit of a tension with the article's 2nd concern: if mere output-mimicking *can't* match more-sophisticated training, then it can't really create the concerning uniformity, either.
And maybe: the dose makes the poison. A little cribbing might be a beneficial partial accelerator for smaller teams & newer projects, even if a lot is ineffective (at ever fully replicating OpenAI model behaviors), or deleterious (if effective and also overdone).
So the article isn't really a strong case for not trying this at all – just for keeping the potential limits & downsides in mind, in any experiments with this technique.
I understand what you mean, and that's a fair point. However, as John Schulman pointed out in his talk, it is possible to clone the behavior of the model, but it won't work to avoid hallucination since the underlying pretrained models are different. If we clone ChatGPT's behavior (by using its output), we'll get the worst of both worlds: weird output coming from its RLHF step AND hallucination.
That seems like an experimentally-testable prediction - that attempts to "clone ChatGPT's behavior (by using its output)" will necessarily "get the worst of both worlds: weird output coming from its RLHF step AND hallucination".
The reasoning that the results won't be quite as good as RLHF, or result in a perfect 'clone' of ChatGPT's capabilities, seems pretty good to me.
But the idea it won't be helpful at all, especially to projects that are just seeking some incremental advantage? Seems speculative.
In particular, when you read the linked comments from Yoav Go, he outlines a potential RL process that uses automated scoring for non-exact similarity to preferred answers. Using known (or even 'probably') good answers from ChatGPT output, as the inputs to that process, seems like it could often offer some of the same sort of improvement to other models as ChatGPT obtained via its RLHF.
The post is referring to fine tuning. You can train your model on the internet, but then it will just produce internet content and not behave subserviently (or carrying the operators desired polical biases rather than random internet ones). Models like ChatGPT take an internet trained model and then perform additional training to make it less internet-like and more cooperative.
Some recent research hash showed reasonable success transferring fine tuning between models using outputs.
Well, they have all of the outputs of ChatGPT stored on their own servers. I suppose it wouldn't be out of the question to filter any future datasets they scrape against the outputs they have.
A watermark is absolutely possible - see for example some of the work Scott Aaronson has mentioned doing for OpenAI.
But: very fragile, especially if people are specifically trying to hide their GPT use, or have access to the watermarking algorithm or online oracle.
And: other methods – like remembering all output ever, or fuzzy summary representations of all output ever – seem to me similarly fragile, & introduce other problems & impracticalities.
A guess: OpenAI internally initially shared the common concern that "consuming its own junk outputs" could be a problem. But their own experiments so far, private & public, may have convinced them it's not as much of a problem in practice as it seems in theory. The model outputs have a mix of good and bad text – just like the pre-LLM internet. And, the same filterings/weightings that have worked on pre-LLM content keep working. And, counter to some early intuitions, often one LLM's quality output is in fact very-useful input for other later LLMs.
I think my comment was misunderstood. I didn’t mean the output text would contain some identifying information. Rather, OpenAI could generate a fingerprint from the text, similar to Apple’s neural has for images, and store that so they can filter out generated text later.
Presumably the ChatGPT content that makes it onto the web is at the very least curated by humans, making that text on average slightly higher quality than the raw output of ChatGPT. If that's the case than you would expect model performance to continue to improve even if the dataset is polluted.
Doesn't matter. We want high-quality text - it's not necessary for it to be human-written. Social signals like upvotes or PageRank will still remain useful even if most text is AI generated.
I certainly don't want most of discussion forums to be generated by bots. I'd rather there was none of it. High-quality generated text is good for fiction and summaries, but not when you want to hear what actual humans have to say.
You just gotta get the AIs to do the upvoting, then cut the humans out of the loop all together and only have AIs read the AI generated text, and then everything will be fine. Just an endless death spiral of ai gen, ai filtering, and ai consumption, forever and ever.
Presumably at some point computers will become (already are for all I know?) the largest consumers of content on the internet as well as its producers.
Bold assumption that AI generated text won't get cheaper exponentially. It already costs less than human generated text of the same quality by magnitudes.
I think you're very confused about the costs required in operating a human... Or are you assuming because the human was going to be doing it anyway the cost is free?
I don't think this problem matters as much as people say it does, except maybe from a research perspective. The chatbot has essentially become part of human culture, it speaks human languages and could actually subtly influence the way human language works. It may develop its own idioms and communication style, and humans may adopt some of this. So yes: now that LLMs are released, everything is polluted in some way, similar to radioactive isotopes. But language is descriptive, not prescriptive: it always works as long as there is shared understanding. People will cherry pick the ChatGPT answers they were able to understand when publishing to the internet, and ignore/ridicule the output that didn't make sense to them.
Note that GPT-3.5 and above are already intentionally polluted with their own output by the RLHF process.
> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs.
To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reason it's usable is because spam filters have gotten pretty good. Even non-trivial amounts of heavily-curated social media (Twitter/FB/IG) is purely spam.
Yeah, I remember reading about things like sports reports and weather being generated by computers ages ago in the likes of SciAm or New Scientist. I don't recall if they used the term AI, I think they did but this was a long time ago.
A couple of financial reporting sites use machines to write articles on small cap stock ticker quarterly reports. It ends up being pretty generic but occasionally is nice to have at a glance human readable summary for when random tiny biotech is suddenly in the news out of nowhere.
I mean, sports reports are on the same level as airport announcements. There is really no need for a person to waste their life away just reporting plain boring numbers.
Spam is already easily generated so AI won't change that.
Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there.
However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud companies create puppet accounts to waste the fraudster's time, and thus the game of cat and mouse continues.
The key difference is that LLMs will allow an unforeseen degree of interactive and custom spam that will feel less and less like spam and more and more like a natural and even useful interaction, until you just realize it was an elaborate scheme to subtly increase your preference for brand A over brand B
Or it will still end up in a bin called statistical spam filter.
It's relatively easy to detect when someone is trying to sell you something. Does not matter if it's a custom written text or not.
We had the true spam filters going 2 nines accurate already. (Google's is somehow misgauging spam or not learning my particular mix of spam well.)
It will be fun to watch the arms race where the spam generator need to conceal prompt injection attacks meant to circumvent such filters while at the same time be too subtle for a humans to pick up
Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago? As far as I know, most of these crappy articles were content farms, often using templates and outsourced labor, but not automatically generated content.
"Can you give an example of any significant scale of fully automatically published blog posts from 10 years ago?"
For some reasons I never bookmarked those sites, when I left them in a rage and disgusted about so much information garbage. So it definitely has become way worse, but also 10 years ago I remember that pattern. Most often when looking for alternatives of software, then you were always a click away of being on a nonsense site, automatically filled with all the relevant keywords and lots of things to accidently click on, but nothing useful. Some of it might have been manual edited, but for the most part, I could allmost see the algorithms that filled those sites up with "content". There were just really primitive - so things will get interesting when this will gets combined with LLMs big scale.
Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT
>For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer.
The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight.
Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference.
Yes, the inner link to the Yoav Go writeup is the gem here, concisely explaining the benefits of RLHF-scoring over mere prediction.
Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities.
My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning.
In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.
It seems pretty transparent, as I think you might be implying in part, that attempting to leap-frog without directly copying training data or model weights is a temporary optimization for “bootstrapping” teams.
I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;)
There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need.
Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part.
Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.
I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking.
Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input.
I know at least one model purposefully removing these utterances, but still. Everyone else is chasing the agent feel, and I'm here pulling my hair out because prosumer were this close to be able to access a proper AI for tasking and now it's slipping away.
Especially, it's not that it can't give you an opinion, it totally can, but the authors at OpenAI don't want you to hear that specific opinion because of their political biases.
Google doesn't have their own LLMs? Microsoft isn't running and training their own copies?
Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).
Schulman's talk is great. I had been thinking about this problem, and he covers almost everything I thought of, with great clarity.
One thing he didn't mention though is that there's potentially a bit of a trick to get the fine-tuning datasets to transfer across models anyway. (I haven't tested it.)
The key idea is to eliminate the pronouns. Imagine asking GPT-4, not whether it knows a fact, but whether "gpt-3.5-turbo" knows, or "text-davinci-003" knows, etc. Then, when you want the model to reply using pronouns (e.g. "I don't know"), use the system message to tell it which model it is.
This doesn't benefit from introspection, so quite possibly it doesn't work. The reason it might work anyway, though, is that estimating the difficulty of a question might be possible even without introspection.
Yes indeed, Yoav Goldberg's post is essentially a good summary of John Schulman's talk, which is excellent. I highly recommend people watch it.
However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don't think it's a good idea or at least it's not enough. Don't get me wrong, it would be really handy to have a free version running on our own cluster, but it wouldn't fix the other issues I mentioned.
It seems more and more plausible that OpenAI chose 2021-09 as a cut-off date was intentional. Because GPT-3 generated output was released into the wild after that.
As others have mentioned, it's frustrating to use a non-OpenAI model and to be told "I'm sorry, as an AI...", as it represents a reimplementation of someone else's censorship.
There are approaches such as Dolly to develop a non-openAI RHLF feedback set but it's hard to compete against ShareGPT and co.
This is not new. We ve been dealing with US standards of morality down to nipples since the beginning of the internet. People will get bored of ChatGPT outputs everywhere, however. We are very good at detecting repeated patterns and tend to find them banal.
There are now uncensored open source models. Vicuna like models are great, and even work for translation. It's eerie what a 10GB file can do
When I finetuned successive generations of ESRGAN on its own output (as I essentially wanted to use it for img2img), it would amplify tiny oddities and artifacts that, I would later find out, were in the training data. Tiny noise splotches, "swirls" and distorted line edges blew up. And I was careful... I pixel peeped the dataset as best I could before starting training.
Human language is obviously different, but I still fear oddities or biases will start popping up when the base models train on large fractions of their own data. And by the time we find out, it will be near impossible to filter out.
But continuing the analogy, maybe a diverse base model population is a good way to avoid that issue?
I think that the article misses the point. Many people are using ChatGPT for creation of relatively small but high quality datasets, because it is very easy. Stanford created an amazing dataset for their Alpaca for just $500.
If you are building a competitive model (such as Meta Llama), then you of course don't use ChatGPT-generated data, because you have the money to download the whole internet.
Yeah, just to be clear, I think using ChatGPT for creating small datasets for niche models makes total sense. I'm talking about creating foundation models which is a different thing.
The author claims to be “flabbergasted” that people would want to stop work on world-changing AI projects.
The gulf between otherwise smart people on this very important issue should depress us all. Personally I feel as if a mutant species has been released into the wild, yet as in Rick and Morty, some people think the wisest course is to release a lot more mutants.
People are fools. Hackers more than most— though we are productive and useful fools much of the time— but it hasn’t been a threat to humanity until recently.
77 comments
[ 2.9 ms ] story [ 143 ms ] threadBut before its concern about a monoculture, the article 1st points out that mere prediction-training (on another model's outputs or fresh data) can't truly match RLHF in instilling some much-desired behaviors.
And that presents a bit of a tension with the article's 2nd concern: if mere output-mimicking *can't* match more-sophisticated training, then it can't really create the concerning uniformity, either.
And maybe: the dose makes the poison. A little cribbing might be a beneficial partial accelerator for smaller teams & newer projects, even if a lot is ineffective (at ever fully replicating OpenAI model behaviors), or deleterious (if effective and also overdone).
So the article isn't really a strong case for not trying this at all – just for keeping the potential limits & downsides in mind, in any experiments with this technique.
The reasoning that the results won't be quite as good as RLHF, or result in a perfect 'clone' of ChatGPT's capabilities, seems pretty good to me.
But the idea it won't be helpful at all, especially to projects that are just seeking some incremental advantage? Seems speculative.
In particular, when you read the linked comments from Yoav Go, he outlines a potential RL process that uses automated scoring for non-exact similarity to preferred answers. Using known (or even 'probably') good answers from ChatGPT output, as the inputs to that process, seems like it could often offer some of the same sort of improvement to other models as ChatGPT obtained via its RLHF.
A subset will give you a subset of the knowledge, it’s no free lunch
Some recent research hash showed reasonable success transferring fine tuning between models using outputs.
[1] - https://en.wikipedia.org/wiki/Multiplicity_(film)
So, not like a watermark, which would be impossible.
But: very fragile, especially if people are specifically trying to hide their GPT use, or have access to the watermarking algorithm or online oracle.
And: other methods – like remembering all output ever, or fuzzy summary representations of all output ever – seem to me similarly fragile, & introduce other problems & impracticalities.
A guess: OpenAI internally initially shared the common concern that "consuming its own junk outputs" could be a problem. But their own experiments so far, private & public, may have convinced them it's not as much of a problem in practice as it seems in theory. The model outputs have a mix of good and bad text – just like the pre-LLM internet. And, the same filterings/weightings that have worked on pre-LLM content keep working. And, counter to some early intuitions, often one LLM's quality output is in fact very-useful input for other later LLMs.
(You can skip to the section “Verifying“)
The rate at which AI can generate text will be so much greater than what humans can generate.
You're already assuming pagerank and upvote systems won't break down in the future.
Presumably at some point computers will become (already are for all I know?) the largest consumers of content on the internet as well as its producers.
Now go ahead and spend $50 dollars on AI generated text nobody is ever going to read, just like almost nobody is going to read this comment.
Note that GPT-3.5 and above are already intentionally polluted with their own output by the RLHF process.
i'd say llm's represent a institutionalized reinforcement of bias (much like journalism) combined with some in-human autonomy.
To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reason it's usable is because spam filters have gotten pretty good. Even non-trivial amounts of heavily-curated social media (Twitter/FB/IG) is purely spam.
Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there.
However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud companies create puppet accounts to waste the fraudster's time, and thus the game of cat and mouse continues.
It's relatively easy to detect when someone is trying to sell you something. Does not matter if it's a custom written text or not. We had the true spam filters going 2 nines accurate already. (Google's is somehow misgauging spam or not learning my particular mix of spam well.)
Open question: is there any case where LLMs can be used for malicious purposes, but LLMs can't be used to defend against it?
It will be fun to watch the arms race where the spam generator need to conceal prompt injection attacks meant to circumvent such filters while at the same time be too subtle for a humans to pick up
Some product comparison websites also seem to be built based on automatic sentence generation from tables with specs.
It's not the same as buzzfeed style content farming but it was a sign of the things to come.
For some reasons I never bookmarked those sites, when I left them in a rage and disgusted about so much information garbage. So it definitely has become way worse, but also 10 years ago I remember that pattern. Most often when looking for alternatives of software, then you were always a click away of being on a nonsense site, automatically filled with all the relevant keywords and lots of things to accidently click on, but nothing useful. Some of it might have been manual edited, but for the most part, I could allmost see the algorithms that filled those sites up with "content". There were just really primitive - so things will get interesting when this will gets combined with LLMs big scale.
>For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer.
The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight.
Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference.
1. https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd...
2. https://www.youtube.com/watch?v=hhiLw5Q_UFg
Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities.
My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning.
In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.
I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;)
There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need.
Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part.
Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.
You can do the same with pure supervised learning and no RL. HF is the key, not RL.
Yoav misses the nuance John had. RL is not bringing something fundamental to the table. It is just a better way to do things at the moment
I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking.
Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input.
I know at least one model purposefully removing these utterances, but still. Everyone else is chasing the agent feel, and I'm here pulling my hair out because prosumer were this close to be able to access a proper AI for tasking and now it's slipping away.
Google doesn't have their own LLMs? Microsoft isn't running and training their own copies?
Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).
One thing he didn't mention though is that there's potentially a bit of a trick to get the fine-tuning datasets to transfer across models anyway. (I haven't tested it.)
The key idea is to eliminate the pronouns. Imagine asking GPT-4, not whether it knows a fact, but whether "gpt-3.5-turbo" knows, or "text-davinci-003" knows, etc. Then, when you want the model to reply using pronouns (e.g. "I don't know"), use the system message to tell it which model it is.
This doesn't benefit from introspection, so quite possibly it doesn't work. The reason it might work anyway, though, is that estimating the difficulty of a question might be possible even without introspection.
However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don't think it's a good idea or at least it's not enough. Don't get me wrong, it would be really handy to have a free version running on our own cluster, but it wouldn't fix the other issues I mentioned.
is there a how to anywhere - not even sure which opensource model to use, etc
There are approaches such as Dolly to develop a non-openAI RHLF feedback set but it's hard to compete against ShareGPT and co.
There are now uncensored open source models. Vicuna like models are great, and even work for translation. It's eerie what a 10GB file can do
When I finetuned successive generations of ESRGAN on its own output (as I essentially wanted to use it for img2img), it would amplify tiny oddities and artifacts that, I would later find out, were in the training data. Tiny noise splotches, "swirls" and distorted line edges blew up. And I was careful... I pixel peeped the dataset as best I could before starting training.
Human language is obviously different, but I still fear oddities or biases will start popping up when the base models train on large fractions of their own data. And by the time we find out, it will be near impossible to filter out.
But continuing the analogy, maybe a diverse base model population is a good way to avoid that issue?
The gulf between otherwise smart people on this very important issue should depress us all. Personally I feel as if a mutant species has been released into the wild, yet as in Rick and Morty, some people think the wisest course is to release a lot more mutants.
People are fools. Hackers more than most— though we are productive and useful fools much of the time— but it hasn’t been a threat to humanity until recently.