Companies like this advocate creating the least secure possible deployments so that they can sell a product that patches some holes they advocated for. Astounding.
What is “Claude’s iMessage integration”? Apple made it? Anthropic did?
An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong?
I feel like everyone is saying 'we're still discovering what LLMs are good at' but it also feels like we really need to get in our collective conscious what they're really, really, bad at.
Another MCP integration mishap demonstrating that Claude can be prompted to go off the rails and can steal, leak or destroy whatever the attacker can tell it to target.
An ever increasing attack surface with each MCP connection.
N + 1 MCP connections + non-determinstic language model + sensitive data store = guaranteed disaster waiting to happen.
Great work. Prompt engineering used for SQL injection style hacking has been predicted long ago, and this is an excellent example of it working in practice. Really hope we pay more attention to this instead of just hyping how agents can change the world. Not so fast.
The "on by default" mitigation is mentioned at the very end:
> Never enable "auto-confirm" on high-risk tools
Maybe some tools should be able to specify to a client to never call it without a human approval.
The security of the MCP ecosystem is basically based on human in the loop - otherwise things can go terribly wrong because of prompt injection and confused clients.
And I'm not sure if current human approval scheme work, because the normalization of deviance is a real thing and humans don't like clicking "approve" all the time...
I feel like we're back in the Windows 98 era. Does nobody remember the days of your local file browser being a web browser? And running native executables in HTML (ActiveX)?? Virtually every PC was getting a virus just plugging into the internet, it was bonkers. Thankfully that plus the DoJ trust busting got Microsoft to back out of all those security nightmares.
And here we are all over again. (double facepalm) I wouldn't touch MCP with a 100-foot pole.
12 comments
[ 2.7 ms ] story [ 20.1 ms ] threadWhat is “Claude’s iMessage integration”? Apple made it? Anthropic did?
- Set up a website without any input sanitization.
- Hey look, you can take control of the database via SQL injection, therefore SQL is completely broken.
- Here's a service you can use to prevent this at your company (which we happen to own).
I feel like everyone is saying 'we're still discovering what LLMs are good at' but it also feels like we really need to get in our collective conscious what they're really, really, bad at.
An ever increasing attack surface with each MCP connection.
N + 1 MCP connections + non-determinstic language model + sensitive data store = guaranteed disaster waiting to happen.
> Never enable "auto-confirm" on high-risk tools
Maybe some tools should be able to specify to a client to never call it without a human approval.
The security of the MCP ecosystem is basically based on human in the loop - otherwise things can go terribly wrong because of prompt injection and confused clients.
And I'm not sure if current human approval scheme work, because the normalization of deviance is a real thing and humans don't like clicking "approve" all the time...
And here we are all over again. (double facepalm) I wouldn't touch MCP with a 100-foot pole.