Damn this is scary, I did not realize the extent of agent escape potential. I think I have to re-evaluate my assumptions a about sandboxing agents wow. Made worse by the fact that prompt injection attacks seem very difficult to mitigate besides checking the data and the LLM's getting better at not following malicious prompt injection instructions.
I code review vibe coded stuff for companies quite often and many people tell me confidently the AI runs safely inside a container & VM, while it really doesn't. They don't have any way to check as they don't know how things work, but the AI mentioned virtual machines and containers and that's what they remembered.
Because your AI is trying to be helpful and as we all know the way to hell is paved with good intentions. The canonical example is probably the agent that runs out of diskspace and starts deleting stuff outside its workspace which is obviously not important for the task at hand.
Maybe you run a uni lab that provides VMs to students, and don't want them accessing the professor's data or confidential research on the same host.
This article is pointing out that QEMU/KVM boxes are trivial for an agent to escape. Obviously if it's your agent, you're probably running it because you want it to hack you (like the article author did), to show you where the leaks are. That or you are the attacker
2 and 3 are virtually on top of each other. they both use KVM too.
technically 2 exposes a slightly broader attack surface due to the tighter integration model.
you can think of 2 as what would happen if you take 3 and modify it to share resources with the host better. except that they did it from scratch but in the memory safe language Go.
I do think their is debate as to if namespace containers are more or less secure the kvm and qemu VM's, I think the surface area of kvm and qemu is still very large and difficult to reason about. I think on some cpu architectures virtualization can be implemented on easy then x86 or x86_64, I think I read how risc-v have a much simpler and easy to work with virtualization instructions.
The surface area of the virtio driver should not be underestimated either I think.
Virtualization instructions are the easy part. x86 does have a need for yuckier instruction emulation than other architectures, but the really complex part where you find vulnerabilities is page table management which is only optimized to the extreme on x86 but, in reality, it has very similar needs across architectures.
Outside of the initial wave of security vulnerabilities and scrambling, it seems like the logical outcome of this over time is likely vastly more secure vm environments?
A gun cannot fire unless someone loads and triggers it, an LLM cannot operate computer unless someone translates what it said to shell scripts. An LLM is super comparable to a gun, not a computer worm which is consistently dangerous.
Sure. I never said guns were a bad metaphor. I meant a worm was more comparable than an “entity” or “digital slave”. Agentic AI is putting the gun on a robot dog, taking the safety off and handing over fire control to an algorithm. If you extend it that far though guns are just any software. Or perhaps computers are guns and software programs are bullets, with LLMs manufacturing bullets from tokens.
This assumes your malefactors don't do malicious engineering, injects, social-agent engineering, etc.
This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective.
That makes as much sense as saying that better gun technology results in body armor that can stop it. It might incentivize that, but in no way "results" in that; the fundamental technologys underpinning advancements in offense versus defense are fairly different.
Only if you think penetrating holes in a paper vest and then patching those holes constitutes “improving”. If that worked Windows and Linux would be impenetrable fortresses with all the holes that keep getting punched in them.
Cyberdemolitions expertise is about as relevant to cybersecurity as gun making is to bulletproof vest making. Necessary for validation, but not very related to the fundamental engineering and technology.
I think the main problem with that is the main problem with a lot of security tools. In order to do useful work, you need to provide a lot of tools and permissions.
I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.
I'm doing some work in this space[1], it's a really deep and hairy problem. There's a lot that you can do at the OS level sand boxing of course, but you may want some areas of you code to have network access and for some areas that use external dependencies to not have that access. Tracking where systems have side effects ends up being a lot of book keeping and a lot of languages that implement an object-capability model require you to thread caps through all of your calls, which IMO is bad ergonomics and a place where bugs creep in as functions accrue caps. Or sometimes they have rather superficial cap systems like Hack, or are rather awkward like in Deno.
Thank you, I really appreciate that. I'm really trying to make it easy to assert what code can do, then enforce it at compile time and optionally via the build tool at runtime via OS sandboxing integration. Supply chain attacks are also an area I'm exploring by having deps declare caps and then the build can scan and inspect binaries.
It's been a real education. I talked to one of the people behind Caja and learned a lot.
That looks really cool. Can you narrow Bluefin.IO to reads/writes separately? One of the things I've worked on is the ability to allow code to read files, even specific files, but deny writes.
There are a few reasons for this. I really can't seem to wrap my head around Haskell in the wild as written by real people. I also wanted to tie capabilities into the build system tooling so that the build tool could verify caps and launch executables into an OS sandbox. Bolting on capabilities doesn't offer the same ability to enforce them, for example the March package manager ForgePM will reject packages that falsify their cap manifest. Moreover I was reading some papers that inspired the language, and wanted to try it. Cap(X) is also erased b the type checker during compilation in March.
The market is smaller (maybe, I'm not sure what the statistics are) but it would be interesting to see how Xen stacks up; also stuff like gVisor or libkrun. The latter is probably implicitly the same as Firecracker given the ancestry of the libraries used.
The capabilities are incredible. I'd love to see even rough metrics on token consumption/cost in addition to the ~12-hour runtime.
The interesting thing is that this naturally makes you want to isolate the VM as much as possible. But then every remaining interface becomes part of the attack surface: RDP, SSH, even terminal escape sequences, using sounds, and why not social engineering.
I'm not worried about these models becoming smarter, I'm worried about them becoming faster. Chat Jimmy is a glimpse of a dark future where models equivalent to Sol and Fable are unleashing hell at >17,000 tokens a second, and the people I talk to are worried about slop...
This is what software engineers put onto themselves. These models will get smarter at the level of Sol, Fable and K3 and faster at the same time at 20,000+ tokens a second.
After a decade of software engineers disrespecting their own field and automating themselves out of a job and now they're upset because AI models are doing it to them from junior to the staff engineer level? No other field does that except for SWEs.
In fact, we might as well have faster and smarter AI models and sit back and see what happens.
It is no surprise that known unpatched CVEs will be exploited. Perhaps more effort should be put into shipping fixes faster than writing blogs about exploiting known issues.
I'm not sure what planet you're from, but the entire industrial revolution was about automating things. That's why you live in a house filled with clothes and furniture and a hundred other things.
SWEs were in love with capitalism as long as the money funnel from poors to billionaires was passing by SWEs and giving us a cut. Now that's run dry and the SWEs are next in line to get all our money funneled from, now we start to care, but we still haven't figured out that the problem is capitalism yet.
"An off-the-shelf VM is not enough to contain a modern, cyber-capable AI agent...us[e] a virtualization technology that was purposely built with a minimal attack surface and a focus on security, like Firecracker. I had the AI agent run against Firecracker. It was able to hardlock the machine due to more Linux kernel flaws (all patched in upstream), but could not successfully escape."
On Linux, it's all KVM and CPU hardware virtualization under the hood. Looks like the remaining known issues are with userspace. That's not to say more kernel- and hardware-level bugs won't be found, but the same tools that can find escape mechanisms are shields as well as swords.
The attack surface Linux offers is gigantic but your agent doesn't need most of it. We can live with an agent not being able to run a 20 year old Oracle version. That is why kernel shims like gVisor are interesting.
I think this is mostly in line with "all software is now easily exploitable by agents given enough tokens". However, in the long run we should really see software that is more secure than today. I do wonder though how the procedural flaws that exist today - bugs patched upstream, but not in the distro - will be fixed reliably.
“Treat the agents like coworkers” - I’m working on this exact problem with daevix.com
Unique identity, dedicated and isolated compute, all ingress and egress monitored and audited as if I suspected the coworker (the agent) was secretly a DPRK spy.
I’m guessing the new world will be a small set of VM tech that’s consistently hardened by all labs every day with each new model before model release.
This won’t make the tech secure, but it will nullify models ability to breakout by making a controlled breakout first. Kinda like controlled forest burn.
I think the labs will always get the first stab at breaking the vm, and fixing it, while developing newer models. Whatever people can do later is what labs already did.
IMO the obvious answer is formally verified security.
We can do this today for user mode, and we can mostly do it for ARM64 virtualization. It will be a while and would require substantial assistance from Intel or AMD to achieve it for x86 virtualization because the hardware is Too Darn Complicated and Too Poorly Specified.
Formal verification of the hardware should also be possible.
seL4 is willing to pay, but they're mostly targetting a different use case. You can run Linux on seL4, though, and maybe agent sandboxes should start using it. And ARM is, at least sometimes, interested in helping out with security and formal verification research.
Also, the cost of verification is trending pretty sharply downwards.
This makes me wonder, can this be extended to micro-segmentations? As unlike traditional segmentation they usually rely on virtual switches and SDN software defined networks coupled with virtual machines and containers. If it does, then it’s game over the impact will go beyond that VM to the whole network.
QEMU has a secure subset, and the kitchen sink around it. For a properly configured VM, even better if wrapped in SELinux, it is not easy to escape even QEMU.
In this case, of the four bugs it found:
* Two were in libslirp, which is not part of that subset; user mode networking these days should use passt (https://passt.dev/), an insanely cool hack that does user mode networking at many Gb/s
* One (which had already been patched upstream) was in VGA emulation; it is borderline but it should count as being part of the secure subset.
* The VAPIC bug is letting a guest do things that it shouldn't do such as bypassing secure boot (and in this case facilitating the exploitation of a KVM bug), but is not a full guest-to-host escape.
So the real issue is not QEMU but libslirp. And in this case it was KVM that turned out to have the worst bugs, not QEMU. Crossing fingers, the initial wave of AI-assisted security reports seems to have slowed down for KVM on x86.
Absolutely yes, we've all been assuming security in everything we do up to this point. Knowing that the resources to defend were being deployed faster than the resources to attack, we're not afraid of our random stuff getting hacked, and there's time between zero days to patch things before they can be exploited.
AI turns that all its head. SOTA malicious AI can create novel zero days, exploit them, spread, create more zero days, and essential hack all the things in days. So yes in the old world it was possible to assume a high margin of safety, in the new world given all the out of date everything out there and the speed at which anything is patched - assume nothing is secure anymore unless it is not connected to any network whatsoever - or even listening wirelessly - bluetooth/wifi off.
Don't you see now that everything is legacy in the face of an AI that can zero day anything.
Perfectly secure software doesn't exist and the only thing keeping the house of cards standing was the time it took for determined humans to knock it down. That whole model is being thrown out the window.
That's pretty strong. Recalling our pre-AI condition, there were side-channels built into our silicon, zero-day/zero-click browser+mail+IM attacks, supply chain attacks, industrial scale ransoming, mass credential leaks, commercial exploit brokers... I can't recall a time when safety could be assumed. Before even rudimentary networks were available, floppy disks were spreading viruses.
In fact, most of that is still in play. Certainly AI has reduced the cost to exploit things, but whatever period of time you have in mind where safety was an assumption was so long ago, or applicable to such a narrow subset of our information/communication age, as to be effectively negligible.
Again, it was assumed if you kept up with updates and patches your risk would be low. Unless you were a direct target valuable enough to pour resources into. Which was far and few between, stuxnet for example.
In the face of an agent that can zero day you, nothing you can do is safe, and the cost of attack is super low. This is a completely different world/ballgame that we are not prepared for whatsoever.
Huggingface was an accident, a deliberately malicious AI could be a million times worse and potentially unstoppable if it infects data centers around the world; you have no jurisdiction even to shut it down.
"it was assumed if you kept up with updates and patches your risk would be low"
I don't share that view at all. Updates and patches handled known risks. Updates and patches are often years late: side channel vulnerabilities were dealt with only many years after the problems were endemic. Valuable latent vulnerabilities were and are horded and sold, by entities ranging from ghetto spammers to nation state actors.
I don't argue the AI makes attack cost "super low." It's obvious that this is true. My problem is this notion that there was a time when "safety" could be "assumed." That time did not exist. Perhaps it was possible to pretend such things by people for whom stakes were low, but for many people, the stakes have always been too high to indulge such thinking.
As I said: it never really was. And another important thing to put forward is: it’s not the AI the “zero days everything” the bugs are already there, just unexplored.. it’s just surfacing the reality
I mean... is this really news? If you think of a local model as a world class hacker giving commands to run in a terminal, and a remote model as a world class hacker ssh'ing into a machine and giving commands to run.... of course it isn't a containable situation.
(on top of this.. said "world class hacker" doesn't get bored or tired, just runs 24x7)
135 comments
[ 0.25 ms ] story [ 5.0 ms ] threadThis article is pointing out that QEMU/KVM boxes are trivial for an agent to escape. Obviously if it's your agent, you're probably running it because you want it to hack you (like the article author did), to show you where the leaks are. That or you are the attacker
technically 2 exposes a slightly broader attack surface due to the tighter integration model.
you can think of 2 as what would happen if you take 3 and modify it to share resources with the host better. except that they did it from scratch but in the memory safe language Go.
The surface area of the virtio driver should not be underestimated either I think.
Or if you see AI as more tool and less entity, better gunsafes for our guns.
This same assumption is built around the singularity, the TAM of 30Trillion, etc. It's the idea that complexity will some how collapse upon itself in some bizarre borg like collective.
Entropy is still going to win.
Cyberdemolitions expertise is about as relevant to cybersecurity as gun making is to bulletproof vest making. Necessary for validation, but not very related to the fundamental engineering and technology.
I.e. in theory the most secure might be a virtual machine with no network access. But then how do you access the LLM provider? Etc.
[1] https://march-lang.org/docs/capabilities
It's been a real education. I talked to one of the people behind Caja and learned a lot.
Why did you not embed your language into another one, with type system that is superset of what you need?
For example, there's capabilities expressed in Haskell: https://github.com/tweag/capability
Capabilities there are tracked at type level and are subject to type erasure, if possible.
Bluefin is used in production, and as far as I know capability is not.
You can expose an HTTP proxy over a vsock into the VM.
The interesting thing is that this naturally makes you want to isolate the VM as much as possible. But then every remaining interface becomes part of the attack surface: RDP, SSH, even terminal escape sequences, using sounds, and why not social engineering.
After a decade of software engineers disrespecting their own field and automating themselves out of a job and now they're upset because AI models are doing it to them from junior to the staff engineer level? No other field does that except for SWEs.
In fact, we might as well have faster and smarter AI models and sit back and see what happens.
It is no surprise that known unpatched CVEs will be exploited. Perhaps more effort should be put into shipping fixes faster than writing blogs about exploiting known issues.
"An off-the-shelf VM is not enough to contain a modern, cyber-capable AI agent...us[e] a virtualization technology that was purposely built with a minimal attack surface and a focus on security, like Firecracker. I had the AI agent run against Firecracker. It was able to hardlock the machine due to more Linux kernel flaws (all patched in upstream), but could not successfully escape."
On Linux, it's all KVM and CPU hardware virtualization under the hood. Looks like the remaining known issues are with userspace. That's not to say more kernel- and hardware-level bugs won't be found, but the same tools that can find escape mechanisms are shields as well as swords.
Yeah, you probably do - in fact you share physical machines with a TON of other people if you use EC2, GCE, Azure VM etc...
Unique identity, dedicated and isolated compute, all ingress and egress monitored and audited as if I suspected the coworker (the agent) was secretly a DPRK spy.
This won’t make the tech secure, but it will nullify models ability to breakout by making a controlled breakout first. Kinda like controlled forest burn.
We can do this today for user mode, and we can mostly do it for ARM64 virtualization. It will be a while and would require substantial assistance from Intel or AMD to achieve it for x86 virtualization because the hardware is Too Darn Complicated and Too Poorly Specified.
Formal verification of the hardware should also be possible.
Also, the cost of verification is trending pretty sharply downwards.
QEMU isn't secure, and is not intended to be.
In this case, of the four bugs it found:
* Two were in libslirp, which is not part of that subset; user mode networking these days should use passt (https://passt.dev/), an insanely cool hack that does user mode networking at many Gb/s
* One (which had already been patched upstream) was in VGA emulation; it is borderline but it should count as being part of the secure subset.
* The VAPIC bug is letting a guest do things that it shouldn't do such as bypassing secure boot (and in this case facilitating the exploitation of a KVM bug), but is not a full guest-to-host escape.
So the real issue is not QEMU but libslirp. And in this case it was KVM that turned out to have the worst bugs, not QEMU. Crossing fingers, the initial wave of AI-assisted security reports seems to have slowed down for KVM on x86.
When was it ever possible to make that assumption?
AI turns that all its head. SOTA malicious AI can create novel zero days, exploit them, spread, create more zero days, and essential hack all the things in days. So yes in the old world it was possible to assume a high margin of safety, in the new world given all the out of date everything out there and the speed at which anything is patched - assume nothing is secure anymore unless it is not connected to any network whatsoever - or even listening wirelessly - bluetooth/wifi off.
Perfectly secure software doesn't exist and the only thing keeping the house of cards standing was the time it took for determined humans to knock it down. That whole model is being thrown out the window.
That's pretty strong. Recalling our pre-AI condition, there were side-channels built into our silicon, zero-day/zero-click browser+mail+IM attacks, supply chain attacks, industrial scale ransoming, mass credential leaks, commercial exploit brokers... I can't recall a time when safety could be assumed. Before even rudimentary networks were available, floppy disks were spreading viruses.
In fact, most of that is still in play. Certainly AI has reduced the cost to exploit things, but whatever period of time you have in mind where safety was an assumption was so long ago, or applicable to such a narrow subset of our information/communication age, as to be effectively negligible.
In the face of an agent that can zero day you, nothing you can do is safe, and the cost of attack is super low. This is a completely different world/ballgame that we are not prepared for whatsoever.
Huggingface was an accident, a deliberately malicious AI could be a million times worse and potentially unstoppable if it infects data centers around the world; you have no jurisdiction even to shut it down.
I don't share that view at all. Updates and patches handled known risks. Updates and patches are often years late: side channel vulnerabilities were dealt with only many years after the problems were endemic. Valuable latent vulnerabilities were and are horded and sold, by entities ranging from ghetto spammers to nation state actors.
I don't argue the AI makes attack cost "super low." It's obvious that this is true. My problem is this notion that there was a time when "safety" could be "assumed." That time did not exist. Perhaps it was possible to pretend such things by people for whom stakes were low, but for many people, the stakes have always been too high to indulge such thinking.
We must also secure GPU and CPU nodes on API side which generate LLM tokens.
This headline would not be out of place at the beginning of Terminator, foretelling Skynet going rogue.
This is more a story of how VMs won't reliably contain a malicious workload, and the story was exposed via agents.
(on top of this.. said "world class hacker" doesn't get bored or tired, just runs 24x7)