A large percentage of my work is peripheral to info security (ISO 27001, CMMC, SOC 2), and I've been building internet companies and software since the 90's (so I have a technical background as well), which makes me think that I'm qualified to have an opinion here.
And I completely agree that LLMs (the way they have been rolled out for most companies, and how I've witnessed them being used) are an incredibly underestimated risk vector.
But on the other hand, I'm pragmatic (some might say cynical?), and I'm just left here thinking "what is Signal trying to sell us?"
This is true. But lately technology direction has largely been a race to the bottom, while marketing it as bold bets.
It has created this dog eat dog system of crass negligence everywhere. All the security risks of signed tokens and auth systems are meaningless now that we are piping cookies, and everything else through AI browsers who seemingly have inifinite attack surface. Feels like the last 30 years of security research has come to naught
This isn't an AI problem, its an operating systems problem.
AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see.
Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either.
Well designed security models don't sell computers/operating systems, apparently.
That's not to say that the solution is unknown, there are many examples of people getting it right.
Plan 9, SEL4, Fuschia, Helios, too many smaller hobby operating systems to count.
The problem is widespread poor taste. Decision makers (meaning software folks who are in charge of making technical decisions) don't understand why these things are important, or can't conceive of the correct way to build these systems.
It needs to become embarrassing for decision makers to not understand sandboxing technologies and modern security models, and anyone assuming we can trust software by default needs to be laughed out of the room.
It's Signal's job to prioritize safety/privacy/security over all other concerns, and the job of an enterprise IT operation to manage risk. Underrated how different those jobs --- security and risk management --- are!
Most normal people probably wouldn't enjoy working in a shop where Signal owned the risk management function, and IT/dev had to fall in line. But for the work Signal does, their near-absolutist stance makes a lot of sense.
This is nothing new, really. The recommendation for MCP deployments in all off-the-shelf code editors has been RCE and storing credentials in plaintext from the get-go. I spent months trying to implement a sensible MCP proxy/gateway with sandbox capability at our company, and failed miserably at that. The issue is on consumption side, as always. We tried enforcing a strict policy against RCE, but nobody cared for it. Forget prompt injection; it seems, nobody takes zero trust seriously. This is including huge companies with dedicated, well-staffed security teams... Policy-making is hard, and maintaining the ever-growing set of rules is even harder. AI provides incredible opportunity for implementing and auditing of granular RBAC/ReBAC policies, but I'm yet to see a company that would actually leverage it to that end.
On a different note: we saw Microsoft seemingly "commit to zero trust," however in reality their system allowed dangling long-lived tokens in production systems, which resulted in compromise by state actors. The only FAANG company to take zero trust seriously is Google, and they get flak for permission granularity all the time. This is a much larger tragedy, and AI vulnerabilities are only cherry on top.
This resonates with what I'm seeing in the enterprise adoption layer.
The pitch for 'Agentic AI' is enticing, but for mid-market operations, predictability is the primary feature, not autonomy. A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability. We are still in the phase where 'human-in-the-loop' is a feature, not a bug.
Recall itself is absolutely ridiculous. And any solution like it is as well.
Meanwhile, Anthropic is openly pushing the ability to ingest our entire professional lives into their model which ChatGPT would happily consume as well (they're scraping up our healthcare data now).
Sandboxing is the big buzzword early 2026. I think we need to press harder for verified privacy at inference. Any data of mine or my company's going over the wire to these models needs to stay verifiably private.
"Hey, you know that thing no one understands how it works and has no guarantee of not going off the rails? Let's give it unrestricted access over everything!" Statements dreamed up by the utterly deranged.
I can see the value of agentic AI, but only if it has been fenced in, can only delegate actions to deterministic mechanisms, and if ever destructive decision has to be confirmed. A good example I once read about was an AI to parse customer requests: if it detects a request that the user is entitle to (e.g. cancel subscription) it will send a message like "Our AI thinks you want to cancel your subscription, is this correct?" and only after confirmation by the user will the action be carried out. To be reliable the AI itself must not determine whether the user is entitled to cancelling, it may only guess the the user's intention and then pass a message to a non-AI deterministic service. This way users don't have to wait until a human gets around to reading the message.
There is still the problem of human psychology though. If you have an AI that's 90% accurate and you have a human confirm each decision, the human's mind will start drifting off and treat 90% as if it's 100%.
> Microsoft is trying to bring agentic AI to its Windows 11 users via Recall. Recall takes a screenshot of your screen every few seconds, OCRs the text, and does semantic analysis of the context and actions.
> Recall takes a screenshot of your screen every
> few seconds, OCRs the text, and does semantic
> analysis of the context and actions. It then
> creates a forensic dossier of everything you
> do into a single database on your computer…
I remember playing around with what sounds like Recall's predecessor back in 2009 [1].
It was only a Microsoft Research project at the time…
>> PersonalVibe
>>
>> Personal Vibe is a prototype
>> Windows Activity Logger that
>> tracks user actions like moving
>> a window or starting an application.
>> The data can be used for a variety
>> of projects from monitoring the
>> actions of study participants to
>> building Vista gadgets that tell
>> you how long you’ve been at work
>> today. The data is stored in a
>> local database that is not remotely
>> accessible. No data is sent from
>> the user’s machine.
>> …
>> Version: 2.0.0.0
>> Date Published: 9 March 2009
Not to take away from the very real security concerns, but this quote is blatantly false to anyone who has used Claude Code or Cursor:
> She said if an AI agent could perform each step with 95% accuracy–which currently isn’t possible–a 10-step task would yield an action with a ~59.9% success rate. And if you had a 30-step task, the success rate would be ~21.4%. Furthermore, if we used a more realistic accuracy rate of 90%, then a 30-step task would drop down to a success rate of 4.2%. She added that the best agent models failed 70% of the time.
I use agentic LLMs (for side projects, properly sandboxed) as much as the next guy, but collectively the normalization of deviance is pretty apparent and shocking (https://en.wikipedia.org/wiki/Normalization_of_deviance).
Backdooring you own machine, sending your .env files, normalizing slop in code review, leaking IP (which is trained on and send to RLHF), YOLO mode, these things would have been unconscionable 2 years ago.
the point about this being an os problem not an ai problem resonates. letting untrusted agents drive your browser smells like a problem to me.
in practice we've had better luck running agents in lightweight sandboxes with explicit capability handles. curious if anyone's tried capability-based systems like sel4 for hosting agents, feels like mainstream oses have a long way to go here.
23 comments
[ 3.2 ms ] story [ 45.3 ms ] threadAnd I completely agree that LLMs (the way they have been rolled out for most companies, and how I've witnessed them being used) are an incredibly underestimated risk vector.
But on the other hand, I'm pragmatic (some might say cynical?), and I'm just left here thinking "what is Signal trying to sell us?"
It has created this dog eat dog system of crass negligence everywhere. All the security risks of signed tokens and auth systems are meaningless now that we are piping cookies, and everything else through AI browsers who seemingly have inifinite attack surface. Feels like the last 30 years of security research has come to naught
Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently.
That's not to say that the solution is unknown, there are many examples of people getting it right. Plan 9, SEL4, Fuschia, Helios, too many smaller hobby operating systems to count.
The problem is widespread poor taste. Decision makers (meaning software folks who are in charge of making technical decisions) don't understand why these things are important, or can't conceive of the correct way to build these systems. It needs to become embarrassing for decision makers to not understand sandboxing technologies and modern security models, and anyone assuming we can trust software by default needs to be laughed out of the room.
Most normal people probably wouldn't enjoy working in a shop where Signal owned the risk management function, and IT/dev had to fall in line. But for the work Signal does, their near-absolutist stance makes a lot of sense.
On a different note: we saw Microsoft seemingly "commit to zero trust," however in reality their system allowed dangling long-lived tokens in production systems, which resulted in compromise by state actors. The only FAANG company to take zero trust seriously is Google, and they get flak for permission granularity all the time. This is a much larger tragedy, and AI vulnerabilities are only cherry on top.
The pitch for 'Agentic AI' is enticing, but for mid-market operations, predictability is the primary feature, not autonomy. A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability. We are still in the phase where 'human-in-the-loop' is a feature, not a bug.
I am pretty much good to go from a it can’t do something I don’t want it to do?
Meanwhile, Anthropic is openly pushing the ability to ingest our entire professional lives into their model which ChatGPT would happily consume as well (they're scraping up our healthcare data now).
Sandboxing is the big buzzword early 2026. I think we need to press harder for verified privacy at inference. Any data of mine or my company's going over the wire to these models needs to stay verifiably private.
I can see the value of agentic AI, but only if it has been fenced in, can only delegate actions to deterministic mechanisms, and if ever destructive decision has to be confirmed. A good example I once read about was an AI to parse customer requests: if it detects a request that the user is entitle to (e.g. cancel subscription) it will send a message like "Our AI thinks you want to cancel your subscription, is this correct?" and only after confirmation by the user will the action be carried out. To be reliable the AI itself must not determine whether the user is entitled to cancelling, it may only guess the the user's intention and then pass a message to a non-AI deterministic service. This way users don't have to wait until a human gets around to reading the message.
There is still the problem of human psychology though. If you have an AI that's 90% accurate and you have a human confirm each decision, the human's mind will start drifting off and treat 90% as if it's 100%.
Good old Microsoft doing microsofty things.
Even recording (which they already are doing) is not exposing sensitive content.
Mix that with hardware enclaves and you actually have a solution to these security and privacy problems.
It was only a Microsoft Research project at the time…
[1] https://g2ww.short.gy/VibeCodeStudioCode> She said if an AI agent could perform each step with 95% accuracy–which currently isn’t possible–a 10-step task would yield an action with a ~59.9% success rate. And if you had a 30-step task, the success rate would be ~21.4%. Furthermore, if we used a more realistic accuracy rate of 90%, then a 30-step task would drop down to a success rate of 4.2%. She added that the best agent models failed 70% of the time.
Backdooring you own machine, sending your .env files, normalizing slop in code review, leaking IP (which is trained on and send to RLHF), YOLO mode, these things would have been unconscionable 2 years ago.
in practice we've had better luck running agents in lightweight sandboxes with explicit capability handles. curious if anyone's tried capability-based systems like sel4 for hosting agents, feels like mainstream oses have a long way to go here.