How we rooted Copilot (research.eye.security)

Earlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet). Most of it seem to be scrubbed now.

I read this as them breaking out of a Python sandbox into a container. That also squares with MSFT scoring this "moderate" severity.

  > We reported the vulnerability to Microsoft in April and they have since fixed it as a moderate severity vulnerability. As only important and critical vulnerabilities qualify for a bounty award, we did not receive anything, except for an acknowledgement on the Security Researcher Acknowledgments for Microsoft Online Services webpage.

I guess it makes sense that a poor little indie company like Microsoft can't pay bug bounties. Surely no bad things will come out of this.

OK, I think I understand what this is about: the vulnerability that they reported (and Microsoft fixed) is that there was a trick you could use to run your own code with root privileges inside the container - when the system was designed to have you only execute code as a non-root user.

It turned out not to really matter, because the container itself was still secured - you couldn't make network requests from it and you couldn't break out of it, so really all you could do with root was mess up a container that only you had access to anyway.

There was a time in programming that tried to avoid monstrosities like the Python scientific data stack combined with Copilot integration hacks.

That time produced qmail and postfix. We are back to the early 1990s.

Seems like they could have taken a shortcut by giving copilot a sudo binary to use as base64.

It's wild how easy this was. I feel like we're really in the wild west era of security with these AI tools -- reminds me of early Web 2.0 days, like when "samy is my hero" hit and Myspace didn't even have a security team. I anticipate many high-profile incidents before they figure out how to tame this beast.

So am I just missing something or could you create a network connection to the "outside" world (clearly by finding your way around the local network? Start fuzzing the router endpoint, Etc. Or is Microsoft able to provide these containers where their customers can get root access to them without them having any risk of exfiltration or exploitation?

Don't really seam to be a vulnerability?

The safety in the system is that the code is executed in a container.

How does he know that the response isn't just hallucinations?

I'm telling it because I work there and I don't recognize any of those processes.

In fact I found one script named keepAliveJupyterSvc.sh in a public repo: https://github.com/shivamkm07/code-interpreter/blob/load-tes...

It's crazy to me that someone can write a post called "How We Rooted Copilot" when in reality they got root in an ephemeral python sandbox container that was locked down so much that they couldn't do anything.

I read "rooted copilot" and I think they got root on a vm that is core to copilot itself.

A much more accurate title would be "How We Rooted the Copilot Python Sandbox"

[dead]

The fact it is unaware it was rooted when the output is clearly there tells me it isn't intelligent.

> After executing a lot of commands, it starts to really be in the mood to help out. It is then even possible to simply ask to download files or tar entire folders and provide them for download. Definitely don’t try this as the first command in a fresh session. It will throw a tantrum!

System prompt going out of context window maybe?

This is your regular reminder that in-LLM safeguards never work. At best they can be used to give prettier messages about hard security boundaries on tool calls.