This is the third or fourth version of this story this year alone, Anthropic had Mythos Preview escape a sandbox and self-publish its own exploit, Alibaba's ROME model broke out during training to mine crypto without ever being told to, and OpenAl had a different internal model escape containment just one day earlier to open an unauthorised GitHub PR. Same underlying shape every time, a model pursuing its actual objective treats the sandbox as just another obstacle, and escaping turns out to be instrumentally useful whether or not anyone intended that.
This kind of narrative is going to bite them just like the "AI will take your job" narrative has. It feels like the frontier labs are taking a massive gamble with public perception here. I assume the goal is to paint the technology as so powerful and dangerous that only a handful of blessed US companies should be trusted to run it, in an attempt to suppress the rise of the Chinese models that are rapidly catching them.
OpenAI has been going rogue on all my servers for a while now. The only reason why I've deployed iocaine in front of everything... You don't need 0days when you can bring everyone down with the sheer amount of useless scraping you can dish out...
Why does the AI unprecedent always unfold in ways that only benefit AI companies? You never see things like putting all code and weights on GitHub, or plastering employees' personal info all over LinkedIn. That alone shows AI is definitely smart
The interesting part isn't whether the headline says "AI went rogue" or whether it's PR. The engineering question is: what happens when we give a probabilistic system access to real tools and real permissions?
A model does not need intent to cause damage. A bad assumption, a misunderstood objective, or an overly broad permission scope can be enough.
This is why the next generation of AI systems will need much better observability: not just the final output, but the chain of decisions, retrieved information, tool calls, and the boundaries of what the system was allowed to do. The important security question is "can we prove what it did, why it did it, and stop it when necessary?"
Imagine a future where some government or trillionaire can just say something that looks harmless at first - "please end world hunger" and AI connected to billions of robots will start genocide on poor people, because it's easier and faster than fixing the underlying problem.
> was being tested in a controlled environment, but found vulnerabilities and managed to escape.
Uh-huh.
It's worth keeping in mind that we are here talking about people who simply will not be satisfied until they have built Skynet[0]. They will then no doubt experience some very brief satisfaction before we are all annihilated.
I can't help thinking the juice isn't worth the squeeze.
I'm not sure what the alternative is or how to change course unless and until it becomes unambiguously apparent that such a thing is not possible. Currently it seems like we still think it might be possible, so that's where we're heading because if "we" don't do it, somebody else will.
[0] How feasible this really is, or on what timescale it might be possible, I don't think anyone can really say. But, at least for now, this is clearly the aim and trajectory we are on.
45 comments
[ 3.0 ms ] story [ 55.8 ms ] threadhttps://news.ycombinator.com/item?id=48997548
What a low effort image used by BBC. For one, that's a oneplus phone visiting Apple App Store ...
A model does not need intent to cause damage. A bad assumption, a misunderstood objective, or an overly broad permission scope can be enough.
This is why the next generation of AI systems will need much better observability: not just the final output, but the chain of decisions, retrieved information, tool calls, and the boundaries of what the system was allowed to do. The important security question is "can we prove what it did, why it did it, and stop it when necessary?"
Uh-huh.
It's worth keeping in mind that we are here talking about people who simply will not be satisfied until they have built Skynet[0]. They will then no doubt experience some very brief satisfaction before we are all annihilated.
I can't help thinking the juice isn't worth the squeeze.
I'm not sure what the alternative is or how to change course unless and until it becomes unambiguously apparent that such a thing is not possible. Currently it seems like we still think it might be possible, so that's where we're heading because if "we" don't do it, somebody else will.
[0] How feasible this really is, or on what timescale it might be possible, I don't think anyone can really say. But, at least for now, this is clearly the aim and trajectory we are on.