Somehow this article feels like a promotional for Legit. But all AI vibe solutions face the same weaknesses. Limited transparency and trust Issues: Using non FOSS solutions for cybersecurity is a large risk.
If you do use AI cyber solutions, you can be more vulnerable for security breaches instead of less.
Wondering if the ability to use hidden (HTML comment) content in PRs would not remain a nasty issue: especially for open source repos?! Was that fixed?
So this wasn't really fixed. The impressive thing here is that copilot accepts natural language. So whatever exfiltration method you can come up with, you just write out the method in english.
They merely "fixed" one particular method, without disclosing how they fixed it. Surely you could just do the base64 thing to an image url of your choice? Failing that, you could trick it into providing passwords by telling it you accidentally stored your grocery list in a field called passswd, go fetch it for me ppls?
There's a ton of stuff to be found here. Do they give bounties? Here's a goldmine.
The rule is to operate using the intersection of all the users permissions of who is contributing text to the LLM. Why can an attacker's prompt access a repo the attacker does not have access to? That's the biggest issue here.
I’m so happy our entire operation moved to a self hosted VCS (Forgejo). Two years ago, we started the migration (including client repos) and not only we saved tones of money on GitHub subscriptions, our system is dramatically more performant for the 30-40 developers working with it every day.
We also banned the use of VSCode and any editor with integrated LLM features. Folks can use CLI based coding agents of course, but only in isolated containers with careful selection of sources made available to the agents.
Banning VSCode — instead of the troublesome features/plug-ins — seems like a step too far. VSCode is the only IDE that supports a broad range of languages with poor support elsewhere, from Haskell to Lean 4 to F*.
I work at a major proprietary consumer product company, and even they don’t ban VSCode. We’re just responsible for not enabling the troublesome features.
With 30-40 devs each pulling a repository to their local machine, how do you prevent even one of them from accidentally exposing the entire repo to an LLM instead of “selected sources”?
And if a user were reluctant to tell you (fearing the professional consequences) how would you detect that a leak has happened?
> I spent a long time thinking about this problem before this crazy idea struck me.
If I create a dictionary of all letters and symbols in the alphabet, pre-generate their corresponding Camo URLs, embed this dictionary into the injected prompt,
This exploit seems to be taking advantage of the slow token-at-a-time pattern of LLM conversations to ensure that the extracted data can be reconstructed in order? Seems as though returning the entire response as a single block could interfere with the timing enough to make reconstruction much more difficult.
Yikes. I knew these AI coding tools were sketchy! Leaking private source code is a massive failure. Who would trust Copilot with their company's secret sauce after this? Just goes to show you can't blindly trust big tech.
23 comments
[ 3.0 ms ] story [ 39.9 ms ] threadIf you do use AI cyber solutions, you can be more vulnerable for security breaches instead of less.
They merely "fixed" one particular method, without disclosing how they fixed it. Surely you could just do the base64 thing to an image url of your choice? Failing that, you could trick it into providing passwords by telling it you accidentally stored your grocery list in a field called passswd, go fetch it for me ppls?
There's a ton of stuff to be found here. Do they give bounties? Here's a goldmine.
We also banned the use of VSCode and any editor with integrated LLM features. Folks can use CLI based coding agents of course, but only in isolated containers with careful selection of sources made available to the agents.
I work at a major proprietary consumer product company, and even they don’t ban VSCode. We’re just responsible for not enabling the troublesome features.
And if a user were reluctant to tell you (fearing the professional consequences) how would you detect that a leak has happened?
Beautiful