I have recently written security-sensitive code using Opus 4. I of course reviewed every line and made lots of both manual and prompt-based revisions.
Cloudflare apparently did something similar recently.
It is more than possible to write secure code with AI, just as it is more than possible to write secure code with inexperienced junior devs.
As for the RCE vector; Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine.
Separately from AI, most people deploy containers based on tagged version names, not cryptographic hashes. This is trivially exploitable by the container registry.
> I have recently written security-sensitive code using Opus 4. I of course reviewed every line and made lots of both manual and prompt-based revisions.
> Cloudflare apparently did something similar recently.
Sure, LLMs don't magically remove your ability to audit code. But the way they're currently being used, do they make the average dev more or less likely to introduce vulnerabilities?
By the way, a cursory look [0] revealed a number of security issues with that Cloudflare OAuth library. None directly exploitable, but not something you want in your most security-critical code either.
Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".
> Is the argument that developers who are less experience/in a hurry,
The CTO of my company has pushed multiple AI written PRs that had obvious breaks/edge cases, even after chastising other people for having done the same.
It's not an experience issue. It's a complacency issue. It's a testing issue. It's the price companies pay to get products out the door as quickly as possible.
Consider the following scenario: You're a developer in charge of implementing a new service. This service interfaces with one of your internal DBs containing valuable customer data. You decide to use a coding agent to make things go a little bit faster, after all your requests are not very complicated and it's bound to be fairly well known.
The agent decides to import a bunch of different packages. One of them is a utility package hallucinated by the LLM. Just one line being imported erroneously, and now someone can easily exfiltrate data from your internal DB and make it very expensive. And it all looks correct upfront.
You know what the nice thing is about actually writing code? We make inferences and reasoning for what we need to do. We make educated judgments about whether or not we need to use a utility package for what we're doing, and in the process of using said utility can deduce how it functions and why. We can verify that it's a valid, safe tool to use in production environments. And this reduces the total attack surface immensely; even if some things can slip through the odds of it occurring are drastically reduced.
If we increase the velocity of changes to a codebase, even if those changes are being reviewed, it stands to reason that the rate of issues will increase due to fatigue on the part of the reviewer.
Consider business pressures as well. If LLMs speed up coding 2x (generously), will management accept losing that because of increased scrutiny?
More and more companies are focusing on costs and timelines over anything else. That means if they are convinced that AI can move things faster and be cost efficient they are going to use more AI and revise cost and time downwards.
AI can write plausible code without stopping. So, not only you get sheer volume of PRs going up at the same time you might be asked to do things "faster" because you can always use AI. I am sure some CTOs might even say - why not use AI to review AI code to make it faster?
Not to mention previously the random people submitting malicious PRs needed to have some experience. But now every script kiddie can get LLMs to out the malicious PRs without knowledge and scale. How is that not a "security nightmare"?
I’ve seen LLMs rolled out in several organizations now and have noticed a few patterns. The big one is that we have less experienced people reviewing code an LLM generated for them. They don’t have the experience to pick out those solutions that are correct 98% of the time, but not now.
When management wants to see dollars, extra reviews are an easy place to cut. They don’t have the experience to understand what they’re doing because this has never happened before.
Meanwhile the technical people complain but not in a way that non technical people can understand. So you create data points that are not accessible to decision makers and there you go, software gets bad for a little while.
Is there a market for apps that use local LLMs? I don't know of many people who make their purchasing decisions based on security, but I do know lawyers are one subset that do.
Using a local LLM isn't a surefire solution unless you also restrict the app's permissions, but it's got to be better than using chatgpt.com. The question is: how much better?
I think the short answer is that there is not one yet, but similar how there is a level of movement behind lan by people that want and can do that, we may eventually see something similar for non-technical users. At the end of the day, security is not sexy, but LLM input/output is a treasure throve if usable information..
I am building something for myself now and local is first consideration, because, as most of us here, likely see the direction publicly facing LLMs are going. FWIW, it kinda sucks, because I started to really enjoy my sessions with 4o
Without a doubt. Companies like Mistral and Cohere (probably others too) will set up local LLMs for your organisation, in fact it's basically Cohere's main business model.
I’ve noticed a strong negative streak in the security community around LLMs. Lots of comments about how they’ll just generate more vulnerabilities, “junk code”, etc.
It seems very short sighted.
I think of it more like self driving cars. I expect the error rate to quickly become lower than humans.
Maybe in a couple of years we’ll consider it irresponsible not to write security and safety critical code with frontier LLMs.
Most of these attacks succeed because app developers either don’t trust role boundaries or don’t understand them. They assume the model can’t reliably separate trusted instructions (system/developer rules) from untrusted ones (user or retrieved data), so they flippantly pump arbitrary context into the system or developer role.
But alignment work has steadily improved role adherence; a tonne of RLHF work has gone into making sure roles are respected, like kernel vs. user space.
If role separation were treated seriously -- and seen as a vital and winnable benchmark (thus motivate AI labs to make it even tighter) many prompt injection vectors would collapse...
I don't know why these articles don't communicate this as a kind of central pillar.
Fwiw I wrote a while back about the “ROLP” — Role of Least Privilege — as a way to think about this, but the idea doesn't invigorate the senses I guess. So, even with better role adherence in newer models, entrenched developer patterns keep the door open. If they cared tho, the attack vectors would collapse.
It'll get better over time. Or, at least, it should.
The biggest concern to me is that most public-facing LLM integrations follow product roadmaps that often focus in shipping more capable, more usable versions of the tool, instead of limiting the product scope based on the perceived maturity of the underlying technology.
There's a worrying amount of LLM-based services and agents in development by engineering teams that haven't still considered the massive threat surface they're exposing, mainly because a lot of them aren't even aware of how LLM security/safety testing even looks like.
Until there's a paradigm shift and we get data and instructions in different bands, I don't see how it can get better over time.
It's like we've decided to build the foundation of the next ten years of technology in unescaped PHP. There are ways to make it work, but it's not the easiest path, and since the whole purpose of the AI initiative seems to be to promote developer laziness, I think there are bigger fuck-ups yet to come.
Why do you think this? the general state of security has gotten significantly worse over time. More attacks succeed, more attacks happen, ransoms are bigger, damage is bigger.
The historical evidence should give us zero confidence that new tech will get more secure.
That's why now I've completely eliminated .env secrets from my codebase and I only use 1Password (with the cli) so it loads secrets dynamically as needed. So if I'm running some AI CLI on my codebase it won't try to leak some secrets
25 comments
[ 1440 ms ] story [ 708 ms ] threadCloudflare apparently did something similar recently.
It is more than possible to write secure code with AI, just as it is more than possible to write secure code with inexperienced junior devs.
As for the RCE vector; Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine.
Separately from AI, most people deploy containers based on tagged version names, not cryptographic hashes. This is trivially exploitable by the container registry.
We have learned nothing from Solarwinds.
> Cloudflare apparently did something similar recently.
Sure, LLMs don't magically remove your ability to audit code. But the way they're currently being used, do they make the average dev more or less likely to introduce vulnerabilities?
By the way, a cursory look [0] revealed a number of security issues with that Cloudflare OAuth library. None directly exploitable, but not something you want in your most security-critical code either.
[0] https://neilmadden.blog/2025/06/06/a-look-at-cloudflares-ai-...
(it won't if you've been following LLM coding space, but anyway...)
I hoped Gary would have at least linked to the talks so people could get the actual info without his lenses, but no such luck.
But he did link to The Post A Few Years Ago Where He Predicted It All.
(yes I'm cynical: the post is mostly on point, but by now I wouldn't trust Marcus if he announced People Breathe Oxygen).
> Refrain from using LLMs in high-risk or safety-critical scenarios.
> Restrict the execution, permissions, and levels of access, such as what files a given system could read and execute, for example.
> Trap inputs and outputs to the system, looking for potential attacks or leakage of sensitive data out of the system.
this, this, this, a thousand billion times this.
this isn’t new advice either. it’s been around for circa ten years at this point (possibly longer).
Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".
The CTO of my company has pushed multiple AI written PRs that had obvious breaks/edge cases, even after chastising other people for having done the same.
It's not an experience issue. It's a complacency issue. It's a testing issue. It's the price companies pay to get products out the door as quickly as possible.
The agent decides to import a bunch of different packages. One of them is a utility package hallucinated by the LLM. Just one line being imported erroneously, and now someone can easily exfiltrate data from your internal DB and make it very expensive. And it all looks correct upfront.
You know what the nice thing is about actually writing code? We make inferences and reasoning for what we need to do. We make educated judgments about whether or not we need to use a utility package for what we're doing, and in the process of using said utility can deduce how it functions and why. We can verify that it's a valid, safe tool to use in production environments. And this reduces the total attack surface immensely; even if some things can slip through the odds of it occurring are drastically reduced.
Consider business pressures as well. If LLMs speed up coding 2x (generously), will management accept losing that because of increased scrutiny?
AI can write plausible code without stopping. So, not only you get sheer volume of PRs going up at the same time you might be asked to do things "faster" because you can always use AI. I am sure some CTOs might even say - why not use AI to review AI code to make it faster?
Not to mention previously the random people submitting malicious PRs needed to have some experience. But now every script kiddie can get LLMs to out the malicious PRs without knowledge and scale. How is that not a "security nightmare"?
When management wants to see dollars, extra reviews are an easy place to cut. They don’t have the experience to understand what they’re doing because this has never happened before.
Meanwhile the technical people complain but not in a way that non technical people can understand. So you create data points that are not accessible to decision makers and there you go, software gets bad for a little while.
It's been going on since Stack Exchange copypasta, and even before that in other forms. Nothing new under the sun.
Using a local LLM isn't a surefire solution unless you also restrict the app's permissions, but it's got to be better than using chatgpt.com. The question is: how much better?
I am building something for myself now and local is first consideration, because, as most of us here, likely see the direction publicly facing LLMs are going. FWIW, it kinda sucks, because I started to really enjoy my sessions with 4o
Without a doubt. Companies like Mistral and Cohere (probably others too) will set up local LLMs for your organisation, in fact it's basically Cohere's main business model.
It seems very short sighted.
I think of it more like self driving cars. I expect the error rate to quickly become lower than humans.
Maybe in a couple of years we’ll consider it irresponsible not to write security and safety critical code with frontier LLMs.
But alignment work has steadily improved role adherence; a tonne of RLHF work has gone into making sure roles are respected, like kernel vs. user space.
If role separation were treated seriously -- and seen as a vital and winnable benchmark (thus motivate AI labs to make it even tighter) many prompt injection vectors would collapse...
I don't know why these articles don't communicate this as a kind of central pillar.
Fwiw I wrote a while back about the “ROLP” — Role of Least Privilege — as a way to think about this, but the idea doesn't invigorate the senses I guess. So, even with better role adherence in newer models, entrenched developer patterns keep the door open. If they cared tho, the attack vectors would collapse.
No current model can reliably do this.
The biggest concern to me is that most public-facing LLM integrations follow product roadmaps that often focus in shipping more capable, more usable versions of the tool, instead of limiting the product scope based on the perceived maturity of the underlying technology.
There's a worrying amount of LLM-based services and agents in development by engineering teams that haven't still considered the massive threat surface they're exposing, mainly because a lot of them aren't even aware of how LLM security/safety testing even looks like.
It's like we've decided to build the foundation of the next ten years of technology in unescaped PHP. There are ways to make it work, but it's not the easiest path, and since the whole purpose of the AI initiative seems to be to promote developer laziness, I think there are bigger fuck-ups yet to come.
The historical evidence should give us zero confidence that new tech will get more secure.