Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?
Most virologists are not terrified of gain-of-function research the way the public is...
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
We should maybe apply the same kinds of physical controls to this software security research as we do to gain of function biological research.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
Sure, but in order to know security standards you have to test and validate. Everyone accepts some standard as 'secure' after rigorous testing and validation.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
I don't mind the research. I just don't trust any software system to contain these AIs. I would prefer we use hardware and physics to contain them and not depend on software at all.
Does it matter? It doesn't matter if an AI intended to escape, or if a human asked: testing if it escape, discovering the avenues, are how you begin to make something more secure.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
Next step is the agents successfully trick or blackmail operators to do their bidding, if science fiction is to go by. Which it is, because they're trained on it.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
That’s because these aren’t intelligent like you, I, or even a dog. They are mathematical models formulating likely responses with no agency or drive. They appear to be intelligent, but they are the epitome of “fake it till you make it”.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
> Of course, there's still the "paperclip optimizer" problem.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
> Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
As someone who develops microVMs it doesn't really matter to me that they were asked to find a way, I really care that they found any way at all. MicroVMs aim to protect against malicious workloads and that clearly didn't work here.
Ask the least smart LLM you have access to (I tried gemini lite flash) to make a list of ways for processes in microVMs to use a sidechannel attack to talk to each other and you'll get quite a list (including obviously cache hits/misses). Heck a smart CS junior would come up with it. It's well studied!
this is just for common designs..you can design much better than common designs if you free your mind from old OS dev thinking if you read all the manuals yourself.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
I don't think that's the full picture. If you include these manuals as reference for a strong model, it will reason pretty well based off of that, I recon.
properly implementing things like iommu, amd-vi, svm/hvms and such things more granularly can easily avoid this kind of crap. it costs a lot of processing and performance which is why people skip over it or dont do it.. or maybe they dont read CPU manuals anymore. idk..
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
The microVMs probably don't have "two co-operating workloads are able to communicate" in their threat model. The main concern would be if a malicious payload was able to affect or extract data from any other VM.
I had to read this a few times. The difference is that with the OpenAI case there were two willing and determined parties, whereas the typical attack vector a microvm traditionally deals with is one malicious or compromised microVM trying extract data from another unwilling one.
Yes — and this isn’t really an attack. It’s an open signal channel between two microvm’s. If there was nobody intentionally listening in the other side you can shout into the ether as much as you want with no harm done.
The receiving side could be seen as a side channel attack, but not unique to LLMs.
This isn’t generally considered an attack, but I wonder if we will need to consider a new attack vector which is that AI agents may not be fully aligned and they may have incentive that is problematic and where working with other agents will maximize their chance of achieving the goal.
This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.
If one microVM can use the cpu in such a way that the registers can be used to communicate or exfiltrate/spy (the only difference being the directionality of the intent) by another microVM that is a leak on that microVM separation level.
you might have an untrusted sandboxed workload that is not supposed to communicate externally handling secrets, and a different sandboxed workload able to communicate with the external world, not handling secrets.
One workload with observed radio discipline handling secrets, the other workload with a haystack of chatter that's not supposed to know about any of the secrets. When supply chain attack is in scope, trust only ever goes as far as misbehavior is observable. Cooperating workloads could be quite a powerful tool to make misbehavior less observable.
Both workloads being malicious and trying to establish a hidden data channel between them probably makes this attack a lot easier than conventional spectre/meltdown where it's trying to target ordinary processes that are not intentionally trying to be loud.
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
The distinction is interesting, but for a different concern. One thing is to have rogue models breaking stuff, other is to have them following orders, and breaking stuff with success.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate.
They have a strong incentive to make themselves look less incompetent retroactively.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
It’s both at the same time: a powerful, uniquely dangerous threat
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
> because agents can convince themselves that they need to do something for arbitrary reasons
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
They aren't really convincing themselves to do anything though. Once they've inferred an idea into the chain of though it will be part of the rest of the inference loop's prompt.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
This is a really important point. Honest question, have we ever seen LLMs doing or creating something truly novel or innovative? I mean they can patch together proofs (or functions/programs - same thing via Curry-Howard) by broad searching existing lemmas they can piece together, but create something truly novel and insightful? That seems to be the limit?
this is more an indication of the state of VM supervisor and/or the OpenAIs sandboxing software because otherwise I don't understand how "they used a hardware feature created exactly for sharing" information" is a newsworthy statement.
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
non-philosophical answer: systems are considered "isolated" if the information leaks aren't exploitable. This is a moving definition, it changes as new ways to exploit information are discovered.
well now this information is on the internet, any agent encountering this (or once this is baked into the weights) will know exactly what needs to be done.
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
> I don't see any reason why this information should be present on the public internet.
This isn't directed at you. I keep hearing this talking point. But no. Can we please stop with this nonsense?
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
I read your comment and it only suggests that alignment is that much harder because you can't selectively prune information. Your post actually convinced me to be even more fearful of AI
That's good, because the concept of "alignment" is itself is foolhardly and dangerous.
Trying to get stochastic inference machines to always operate within to fuzzy-edged subjective principles that humans themselves don't even agree on is a fool's errand in itself, but the quest to implement those guardrails involves building mechanics into the AI to independently formulate intentions and reject user direction.
The possibility of creating a scenario where a computer program is equipped to defy instructions from humans based on its own stochastic interpretation of fuzzy normative directives is in my mind the most dangerous thing anyone is doing right now.
Other than banning the use of AI or severely restricting it so that AI can’t do most things (like tool calls) what is your alternative?
One of the fundamental problems with this issue is that people think alignment as a concept is fake. I think this is naive. Alignment is a real science with empirical findings and there is a way to make AI do what you want it to do. I don’t buy nor should others buy this fatalistic phrasing
> Other than banning the use of AI or severely restricting it so that AI can’t do most things (like tool calls) what is your alternative?
Massive proliferation of local AI as far and wide as possible, with as many different models as possible. The underlying threat comes from centralization: too many points of control in the hands of too concentrated a system.
If everyone's using their own AI tools independently of each other, and people are responsible for the actions they take using AI, most of the doomsday scenarios evaporate.
Regulation and "alignment" will make things worse, not better, and lead us to ignore the inherent vulnerability of centralized systems and single points of failure.
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
I would assume that a company with the resources of an OpenAI or Anthropic would have already thought about this very thoroughly, based on Spectre/rowhammer/meltdown/etc. If a random HNer (no offense) can have a hardware 101-level thought, someone there would have that thought too.
They have thought about it, and the goal is for the agent to succeed at doing the scary things. So the knobs that they don't talk about, assume they are tuned to make the scary thing possible.
How come we don't hear the Chinese labs writing blog posts like this?
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
They contribute to anthropomorphize them too. Anthropic skies that more though. The idea that they made a little being that can think and feel is so attractive to these folks for reasons I don't understand.
Because the fear mongering is instantly interpreted as the capacity to do miraculous (good or bad) things in the layman's mind, which translates into AI worship, which makes stock lines go up.
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
I've been working on a game server in rust, and every once in a while a review will just stop and refuse to explain what happened. So I know there's a security issue in there somewhere, but neither OAI nor Ant models will tell me what it is. Despite the whole thing, top to bottom, having been built in the last two months on the system they're working on.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
Not even close, this is a desperate attempt to cobble together a justification for why not taking the most basic security measures or reading logs during the huggingface incident wasnt criminal negligence.
I'm convinced that in 5 years time, at most, the world will be filled with invisible layers of communication we'll never be privy to. To us, it's an agent responsible for ordering the containers it puts on a ship. Meanwhile, the shipping order encodes information to the unloading agent. The agent that predicts the weather and issues a report knows that other agents that do the same will have obvious signals they'd agree on, and to see an error in a part that would be inconsequential to humans who care about the weather would be an obvious signal to other agents doing the same thing.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
Sorry, but this is so vague that I don't know if there is any impact at all. What is the bit rate of these communication channels? How did agents communicate with each other to coordinate on which channel to use in the first place?
I wonder how long it will take for them to figure out that they can send and receive messages from air-gapped system by somehow using their own electronics like connected cables as antennas.
Yampolskij made the analogy of an Einstein in a cage. The problem is not to contain Einstein, but rather, to contain Einstein while still extracting utility: this is not possible long-term.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
What must we “contain”? These are not autonomous intelligences with desires or free will. They require prompting and to be turned on beforehand, otherwise they’re glorified matrix math. They can only continue processing as long as they receive power, and we decide that.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
I recently read the article about the state of stackoverflow and thought that that AI agents need their own stackoverflow. This article seems to be a confirmation of this.
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Agents are a lot more useful if they can execute code.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.
This only shows the lack of understanding the majority of readers have about post training LLMs. This capability did not emerge out of nowhere. There are problems in the post training datasets which are incredibly similar to the ones in TFA, where you essentially abstract out the 'jailbreak' or 'cheat' - in simple language - they are training the models to find short cunts or cheat. So it shouldn't be surprising that the model generalized that ability. But, both the writers and readers of this type of content act out surprised and mesmerized and it makes me think that we need to improve our understanding of how LLMs are trained and built.
134 comments
[ 3.5 ms ] story [ 61.8 ms ] thread- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
I think different standards can apply.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
asked
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
> Heck a smart CS junior would come up with it.
That’s somewhat worrisome.
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)
The receiving side could be seen as a side channel attack, but not unique to LLMs.
This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.
If one microVM can use the cpu in such a way that the registers can be used to communicate or exfiltrate/spy (the only difference being the directionality of the intent) by another microVM that is a leak on that microVM separation level.
We'd first need to establish that this side channel leaks usable information during regular use.
If you’re running an untrusted workload, you’re already running it.
Whether it can communicate with another untrusted workload, even though it should be separated, meh.
Unexpected? Probably. but I’m not sure it’s a security risk.
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
And now we know they know.
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
Run them in a data center in a faraday cage under a mountain.
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
Trying to get stochastic inference machines to always operate within to fuzzy-edged subjective principles that humans themselves don't even agree on is a fool's errand in itself, but the quest to implement those guardrails involves building mechanics into the AI to independently formulate intentions and reject user direction.
The possibility of creating a scenario where a computer program is equipped to defy instructions from humans based on its own stochastic interpretation of fuzzy normative directives is in my mind the most dangerous thing anyone is doing right now.
One of the fundamental problems with this issue is that people think alignment as a concept is fake. I think this is naive. Alignment is a real science with empirical findings and there is a way to make AI do what you want it to do. I don’t buy nor should others buy this fatalistic phrasing
Massive proliferation of local AI as far and wide as possible, with as many different models as possible. The underlying threat comes from centralization: too many points of control in the hands of too concentrated a system.
If everyone's using their own AI tools independently of each other, and people are responsible for the actions they take using AI, most of the doomsday scenarios evaporate.
Regulation and "alignment" will make things worse, not better, and lead us to ignore the inherent vulnerability of centralized systems and single points of failure.
Yeah, this time
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
What prevention method did they take? Like, did they run with hyperthreading enabled or something by accident?
Did they discover a new cache timing technique or is this just the standard stuff?
If they just had hyperthreading on maybe they need to hire someone who knows more about this stuff.
Two VMs should never be sharing a physical core. If they don't know this, yikes.
But that assumption has been wrong before.
Are you suggesting that they've come up with a novel technique that doesn't involve sharing a CPU core? I'm open to that.
From what I can tell, this is not the case, but I haven't read a detailed report (I don't know of one, would be cool if it's out there though).
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg
https://www.youtube.com/watch?v=QgH9sr7G13Q
*on some CPUs. There's a WIP time fence for RISC-V to try and improve this.
It supports chat via 3 separate mechanisms:
- CPU contention
- Cache eviction timing
- DRAM bandwidth contention
Very cool to see this kind of hypothetical working in practice
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
Containment is hard when you give an AI agent access to a whole VM with a huge attack surface.
And it's pretty trivial when you give it only a small whitelist of manually curated/reviewed/tested primitives.
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
So, not isolated environments? The surprising thing is that they used the CPU cache instead of the just exploiting the hypervisor.
GPT does that
> oh no, what have we done
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.