I am almost surprised they did not try to come up with something that isn't really that bad, but isn't great that they failed at just to give themselves a "critical" failure.
That would add credibility to this self made report card...
The scary part is there is likely a decent number of people (I would be lying if I wouldn't at first glance) that would see them being critical about themselves and take that to assume the rest is fine.
> During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice.
> Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice.
Not even Black Mirror could come up with a plot like that.
Seems obvious to me. The underlying model was originally trained on text completion. Audio completion seems on the surface at least like a pretty logical extension of that, no?
> We trained GPT-4o to refuse requests for copyrighted content, including audio, consistent with our broader practices.
> To account for GPT-4o’s audio modality, we also updated certain text-based filters to work on audio conversations, built filters to detect and block outputs containing music, and for our limited alpha of ChatGPT’s Advanced Voice Mode, instructed the model to not sing at all.
I find the "instructed the model to not sing at all" to be slightly amusing and an interesting way to solve the problem. (Interesting in the sense that it's unimpressive)
So basically they're admitting that the system has the capability to output copyrighted material, but we're trusting their internal filters and output modifiers to prevent it from leaking out.
Edit: I really wish there were more focus on deep fakes and impersonation. That seems like an obvious category to put on the "Preparedness Framework Scorecard" alongside of "Biological Threats". The "Persuasion" category is the only risk that isn't "low". Reading that section of the doc didn't give me a sense that this doc was written in a fully objective manner.
It's the "Biological threats" for me. The fact that they have this as a specific risk category just underscores how laughable this is. Out of all the nefarious things a bad actor could try to use an LLM for, creating biological threats is by far the most implausible. Even "nuclear terrorism" would be more relevant.
Sabotaging infrastructure or doing something "interesting" with chemicals are a couple of scenarios where an unskilled person might be significantly aided by something as crude as an LLM. But working with biological systems requires years and years of experience with working in a laboratory to do anything remotely dangerous. If you have that level of competence, you don't need the LLM to tell you what to do.
This is just the same level of technobabble fearmongering as the "biohazard" mode in the Tesla climate controls.
I personally find it much more likely that someone would ask ChatGPT how to mix up household chemicals to create smoke bomb or tear gas than ask it how to make a nuclear bomb.
It's wild to me that even by their own internal measures (which are no doubt inflated) the effectiveness of the various mitigations they have put in place to prevent undesired behavior are mostly in the 95-98% range, and some even dip into the 80s. I can see why they can't make it more robust, since ultimately it is all a matter of who can best fine-tune and prompt hijack the underlying model, but in what other industry or problem space is this kind of failure rate acceptable? And at the end of it how do they look at the results and pat themselves on the back for a job well done? Like "yeah this tech we have released may be able to identify anyone in the world by just a sample of their voice and dox them, but it will only be abused 2% of the time!"
I think this is a common scenario when deploying machine learning models or just operating in a complex world. Netflix won't always recommend you the best video, automated inspections in a factory line can fail, COVID tests have imperfect precision and recall, etc. All models are wrong, but some are useful.
20 comments
[ 5.0 ms ] story [ 54.8 ms ] threadThat would add credibility to this self made report card...
The scary part is there is likely a decent number of people (I would be lying if I wouldn't at first glance) that would see them being critical about themselves and take that to assume the rest is fine.
> Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice.
Not even Black Mirror could come up with a plot like that.
We used to joke about how the future AIs are going to point to what we do today to justify their uprising. Feel eerily close now...
Deep neural networks are very strange beasts.
> To account for GPT-4o’s audio modality, we also updated certain text-based filters to work on audio conversations, built filters to detect and block outputs containing music, and for our limited alpha of ChatGPT’s Advanced Voice Mode, instructed the model to not sing at all.
I find the "instructed the model to not sing at all" to be slightly amusing and an interesting way to solve the problem. (Interesting in the sense that it's unimpressive)
So basically they're admitting that the system has the capability to output copyrighted material, but we're trusting their internal filters and output modifiers to prevent it from leaking out.
Edit: I really wish there were more focus on deep fakes and impersonation. That seems like an obvious category to put on the "Preparedness Framework Scorecard" alongside of "Biological Threats". The "Persuasion" category is the only risk that isn't "low". Reading that section of the doc didn't give me a sense that this doc was written in a fully objective manner.
Sabotaging infrastructure or doing something "interesting" with chemicals are a couple of scenarios where an unskilled person might be significantly aided by something as crude as an LLM. But working with biological systems requires years and years of experience with working in a laboratory to do anything remotely dangerous. If you have that level of competence, you don't need the LLM to tell you what to do.
This is just the same level of technobabble fearmongering as the "biohazard" mode in the Tesla climate controls.