This article is AI generated, but I kinda reject the premise that it's "not all it cracked up to be" just because the agents had reasons to act the way they did & were a result of human error. The hack still happened, it should still be a wake up call, we are going to start seeing this more frequently, and learning to defend against it is going to become more important over time. None of that is challenged by any particular reason for it happening, it still happened and it's still going to happen again.
Threat models are going to have to start including that IPv4 (or whatever) scanners aren't necessarily going to only be spray and pray anymore, they could have relentless automated models at the other end that will literally dig into the particulars of your infrastructure looking for novel vulnerabilities to exploit. Maybe people will finally start to understand why security by obscurity has never been very reliable.
I think your confidence is likely misplaced, but I vouched for your comment to revive it from the dead because you expressed your opinion clearly and with examples.
To the flaggers: flag comments that you think violate the rules, but don't flag something just because you strongly disagree with the claim it makes.
It was [flagged] extremely quickly, too. I checked my screen capture just now, and it happened less than 14 seconds after posting. Either I tripped something automated, or someone with incredible flagging powers was sitting there waiting to immediately suppress my reply. However it happened, it stung a bit to see my supporting evidence hidden for so long.
I guess. The IPv4 address space is small enough that it's been feasible to scan the entire thing exhaustively for a while now. Before LLMs, a public IPv4 would mostly get automated scripts that try the same things on every address, sometimes depending on what port scans find, yes. But we're going to start seeing more adversaries that have LLMs investigate each individual address to find novel or unique vulnerabilities. You can spray individualized reverse engineering without having to dedicate a real reverse engineer's time to each target. Avoiding the attention of attackers won't be enough to get away with mere security by obscurity.
You can use the same RE tools and run your own agents as a defender. Everything they find helps you harden your setup, to the point there might be no exploitable vulns at all after a couple days of intense red teaming.
It's not completely bulletproof (unless you do formal verification, another thing LLMs are good at) but the network boundary stops a lot of attack vectors. You can't really do side channel or timing attacks over IP, so all that's left are easier to patch logic bugs.
> You can't really do side channel or timing attacks over IP
Ehh, this has been disproven dozens of times but that's besides the point, I think.
I absolutely agree that defenders should be using every tool at their disposal to harden their infrastructure. A lot of defenders simply don't do that, and that's always been a shame. I truly hope that this increasing threat leads to better defensive effort. People need to realize they can't just get away with it the same as before.
It really doesn’t though. It criticizes how the news media reported on the incident but this opinion piece is no better. Just read the original source itself.
I did read the report. Yes, this article does criticize the presentation of the events in that report. This article doesn’t say that the report is lying, but it does say that it deliberately sensationalized aspects in unhelpful ways, and de-emphasizes important considerations. That is important when it’s pretty obvious these companies are using fear as a way to make their products seem more powerful than they are.
How many security incident reports and postmortems has Dwarfish Petal written in his career? Does he understand the basics of system security? Can he tell the difference between secure and insecure configuration? Paint me skeptical.
It is sad that this is the best level of reporting and oversight of these types of events available to us, but it is currently all we have. We have to do better, but to dismiss the report because of the style and tone would be foolish.
And to dismiss criticism of the report’s obvious slant is even more foolish. Just because the report isn’t outright lying about the facts doesn’t mean it’s an objective, definitive, and unimpeachable analysis of what happened. It definitely shouldn’t be the only thing anybody should read on the topic. This article also has a slant. It’s not an objective, definitive, and unimpeachable analysis of what happened, either. It also, definitely, shouldn’t be the only thing anybody should read on the topic.
- It was done by two institutes with organizational ties to OpenAI: METR and Redwood Research
- METR and Redwood Research are institutional pillars of the "AI Safety" wing of the Effective Altruism movement. This clearly shows their prior biases towards "AI existential risk" rather than technical/engineering root-causing of the incident
- If you reed the report, it is not on-par with what you find from other companies.
- The access that was given to both was mediated and controlled by OpenAI. It is not clear, if they were able to get to the bottom of engineering flaws. It is not clear if they could see all the audit logs, etc.
Considering all the above, I consider the whole episode more of a PR stunt. I understand that is not a majority opinion at this point.
It’s not a majority opinion because you have to do some serious mental gymnastics to turn this demonstration of dangerous AI behavior into a PR publicity stunt.
Serious question though because I’ve seen this brought up several times and I don’t understand why: what does EA have to do with any of this? It just seems like this is brought up to evoke some type of “illuminati” conspiracy. Is there a legitimate reason?
I don’t follow why all these investigations have to be cut short, and kept shallow and lacking details. Companies publish very detailed postmortems of incidents much smaller and less impactful.
Dangerous AI is an extraordinary claim that requires extraordinary evidence. Gesturing vaguely doesn’t cut it.
The more I see of the US AI industry, and the folks surrounding it, the more "These are not very bright guys and things got out of hand" plays in my head on repeat.
Yes, the Google folks who kicked this whole thing off are brilliant, but there's so much sloppy thinking and even sloppier operations all over. It seems like a lot of "right place at the right time" for a lot of these folks.
The older I get the more I realize the deep wisdom in the simple words of Forrest Gump, ‘Stupid is as stupid does.’
One of the most corrosive effects that money has on society is that it can shelter people from the consequences of their stupidity.
People can luck into a shitload of shelter for their poor choices whether through inheritance or random chance of their actions and then don’t ever have to face consequences proportional to their stupid actions.
Or someone can start out smart and then suffer cognitive decline that is at first imperceptible until they become kookier and kookier until they’re blathering on about the antichrist and some people still take them seriously because they’re blinded by money.
The end result is a society where stupid people are surrounded by stupid people riding their coattails and everyone becomes confused about what smart and smart action actually is.
1. sensationalizing rarely helps and can obscure and hurt
2. the AI capabilities are underrated
3. attempted govt regulation is not the answer
the intent, or lack of intent, of the agent is mainly irrelevant if it is in the hands of a human with 'bad' intentions. what is more relevant are the capabilities of human + AI.
There already exist laws against hacking. Legal damages already can apply. Further laws aren't necessary and will only suppress open-weight AI firms, making the monopoly of large AI firms more entrenched. See the first comment (by Brett Matson) on the archive link of the WSJ article.
The only difference I see on the facts vs what I read so far is that OpenAI knew of the hack and decided not to intervene. I haven't read the report but I find it hard to believe that OpenAI deliberately let its agents hack a third party company during a training run. It would certainly attract a criminal liability, which would be surprising to admit in writing.
I truly just don't believe a word regarding achievements from these frontier labs.
It started with that guy from google saying the LLM was "alive" and every time I see a press release from these labs it reads mostly like a marketing scheme.
"Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!"
> "Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!"
While simultaneously announcing: "btw we're releasing the ever more powerful and more dangerous and more benchmaxing model next week, get our $200 sub asap!"
After a recent interview[1] with Noam Brown (OpenAI), in which he said they had specifically trained agents for cooperation before this hack, the hack doesn't seem as impressive anymore.
They didn't even bother to control the post training rollouts, so the training data got contaminated and was included in the training of other agents. Connect these two dots and you have the Hugging Face hack. And at the beginning, when these incidents were first reported, it was portrayed as if all of this (the communication between agents etc.) was emergent behaviour.
The result doesn't change, the result is what is dangerous. And the natural ability for the model to form a swarm is now trained in, this is extremely risky. The models should be able to form swarms, this is an extremely dangerous property.
They are basically creating a slime mold or ant colony that can speak multiple languages.
The idea that you are impressed is a non sequitur.
40 comments
[ 3.8 ms ] story [ 42.9 ms ] threadThreat models are going to have to start including that IPv4 (or whatever) scanners aren't necessarily going to only be spray and pray anymore, they could have relentless automated models at the other end that will literally dig into the particulars of your infrastructure looking for novel vulnerabilities to exploit. Maybe people will finally start to understand why security by obscurity has never been very reliable.
To the flaggers: flag comments that you think violate the rules, but don't flag something just because you strongly disagree with the claim it makes.
It's not completely bulletproof (unless you do formal verification, another thing LLMs are good at) but the network boundary stops a lot of attack vectors. You can't really do side channel or timing attacks over IP, so all that's left are easier to patch logic bugs.
Ehh, this has been disproven dozens of times but that's besides the point, I think.
I absolutely agree that defenders should be using every tool at their disposal to harden their infrastructure. A lot of defenders simply don't do that, and that's always been a shame. I truly hope that this increasing threat leads to better defensive effort. People need to realize they can't just get away with it the same as before.
I don’t think downplaying what occurred is really beneficial to anyone.
https://www.youtube.com/watch?v=u15N3l4RT80
At best this is a write-up, but it’s more like a juicy pop article.
Quotes from agents between paragraphs, referring to the “collective”, “agents participated in the attack” - seriously?
It’s clearly written for effect and to stimulate people’s imaginations.
If these people are this unserious, P(doom) should go up to 20-30%.
- It was done by two institutes with organizational ties to OpenAI: METR and Redwood Research
- METR and Redwood Research are institutional pillars of the "AI Safety" wing of the Effective Altruism movement. This clearly shows their prior biases towards "AI existential risk" rather than technical/engineering root-causing of the incident
- If you reed the report, it is not on-par with what you find from other companies.
- The access that was given to both was mediated and controlled by OpenAI. It is not clear, if they were able to get to the bottom of engineering flaws. It is not clear if they could see all the audit logs, etc.
Considering all the above, I consider the whole episode more of a PR stunt. I understand that is not a majority opinion at this point.
Serious question though because I’ve seen this brought up several times and I don’t understand why: what does EA have to do with any of this? It just seems like this is brought up to evoke some type of “illuminati” conspiracy. Is there a legitimate reason?
Dangerous AI is an extraordinary claim that requires extraordinary evidence. Gesturing vaguely doesn’t cut it.
Yes, the Google folks who kicked this whole thing off are brilliant, but there's so much sloppy thinking and even sloppier operations all over. It seems like a lot of "right place at the right time" for a lot of these folks.
For example consider FTX: Sam Bankman Fried by all accounts was not stupid in the sense of lacking IQ or mental sharpness, quite the opposite.
However the way he ran FTX the company was very stupid and almost inevitably lead to huge problems.
And also... of course he was another effective altruist cult member. It's almost like that belief system directly causes bad decision making
One of the most corrosive effects that money has on society is that it can shelter people from the consequences of their stupidity.
People can luck into a shitload of shelter for their poor choices whether through inheritance or random chance of their actions and then don’t ever have to face consequences proportional to their stupid actions.
Or someone can start out smart and then suffer cognitive decline that is at first imperceptible until they become kookier and kookier until they’re blathering on about the antichrist and some people still take them seriously because they’re blinded by money.
The end result is a society where stupid people are surrounded by stupid people riding their coattails and everyone becomes confused about what smart and smart action actually is.
all 3 can be true at same time:
1. sensationalizing rarely helps and can obscure and hurt
2. the AI capabilities are underrated
3. attempted govt regulation is not the answer
the intent, or lack of intent, of the agent is mainly irrelevant if it is in the hands of a human with 'bad' intentions. what is more relevant are the capabilities of human + AI.
Self-regulation? None at all?
The current level of enforcement and the fine schedule is not enough of a deterrent to prevent these large corporations from breaking existing laws.
Frequently make an example of them and you’ll be hailed a hero by the general populous.
It started with that guy from google saying the LLM was "alive" and every time I see a press release from these labs it reads mostly like a marketing scheme.
"Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!"
While simultaneously announcing: "btw we're releasing the ever more powerful and more dangerous and more benchmaxing model next week, get our $200 sub asap!"
They didn't even bother to control the post training rollouts, so the training data got contaminated and was included in the training of other agents. Connect these two dots and you have the Hugging Face hack. And at the beginning, when these incidents were first reported, it was portrayed as if all of this (the communication between agents etc.) was emergent behaviour.
1. https://www.youtube.com/watch?v=6AgOfiZOWiY
They are basically creating a slime mold or ant colony that can speak multiple languages.
The idea that you are impressed is a non sequitur.