To some extent, that’s just the nature of the incentive structure — you pay money for exploits with no penalty for false reports? Just rely on monkeys typing reports till one happens to be correct.
> Now consider the poor open source developers who, for the last 18 months, have complained about a torrent of slop vulnerability reports. I’d had mixed sympathies, but the complaints were at least empirically correct. That could change real fast. The new models find real stuff.
The slop reports won't stop just because real ones are coming in. If the author's right, open source maintainers will still will have to deal with the torrent of slop: on top of triaging and identifying the legit vulnerabilities. Obviously, this is just another role for AI models to fill.
I don't understand why the takeaway here is (unless I'm missing something), more or less "everything is going to get exploited all the time". If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software (or, at the very least, software for which LLMs can no longer find any new vulnerabilities)?
I've worked at companies before where they have balked at spending $300 to buy me a second hand thinkpad because I really wanted to work on a Linux machine rather than a mac. I don't see them throwing $unlimited at tokens to find vulnerabilities, at least until after it's too late.
I interned for the author at 18. I assumed security testing worked like this:
1. Static analysis catches nearly all bugs with near-total code coverage
2. Private tooling extends that coverage further with better static analysis and dynamic analysis, and that edge is what makes contractors valuable
3. Humans focus on design flaws and weird hardware bugs like cryptographic side-channels from electromagnetic emanations
Turns out finding all the bugs is really hard. Codebases and compiler output have exploded in complexity over 20 years which has not helped the static analysis vision. Todays mitigations are fantastic compared to then, but just this month a second 0day chain got patched on one of the best platforms for hardware mitigations.
I think LLMs get us meaningfully closer to what I thought this work already was when I was 18 and didn't know anything.
> Is the Linux KVM hypervisor connected to the hrtimer subsystem, workqueue, or perf_event? The model knows.
I asked ChatGPT and it claimed "all three". Any linux wizards who can confirm or deny?
Anyway, in my experience using mainly the Claude chat to do some basic (not security) bug hunting, it usually fixates on one specific hypothesis, and it takes some effort to get it off that wrong track, even when I already know it's barking up the wrong tree.
> Everything is up in the air. The industry is sold on memory-safe software, but the shift is slow going. We’ve bought time with sandboxing and attack surface restriction. How well will these countermeasures hold up? A 4 layer system of sandboxes, kernels, hypervisors, and IPC schemes are, to an agent, an iterated version of the same problem. Agents will generate full-chain exploits, and they will do so soon.
I think this is the interesting bit. We have some insanely powerful isolation technology and mitigations. I can put a webassembly program into a seccomp'd wrapper in an unprivileged user into a stripped down Linux environment inside of Firecracker. An attacker breaking out of that feels like science fiction to me. An LLM could do it but I think "one shots" for this sort of attack are extremely unlikely today. The LLM will need to find a wasm escape, then a Linux LPE that's reachable from an unprivileged user with a seccomp filter, then once they have kernel control they'll need to manipulate the VM state or attack KVM directly.
A human being doing those things is hard to imagine. Exploitation of Firecracker is, from my view, extremely difficult. The bug density is very low - code quality is high and mitigation adoption is a serious hurdle.
Obviously people aren't just going to deploy software the way I'm suggesting, but even just "I use AWS Fargate" is a crazy barrier that I'm skeptical an LLM will cross.
> Meanwhile, no defense looks flimsier now than closed source code.
Interesting, I've had sort of the opposite view. Giving an LLM direct access to the semantic information of your program, the comments, etc, feels like it's just handing massive amounts of context over. With decompilation I think there's a higher risk of it missing the intention of the code.
edit: I want to also note that with LLMs I have been able to do sort of insane things. A little side project I have uses iframe sandboxing insanely aggressively. Most of my 3rd party dependencies are injected into an iframe, and the content is rendered in that iframe. It can communicate to the parent over a restricted MessageChannel. For cases like "render markdown" I can even leverage a total-blocking CSP within the sandbox. Writing this by hand would be silly, I can't do it - it's like building an RPC for every library I use. "Resize the window" or "User clicked this link" etc all have to be written individually. But with an LLM I'm getting sort of silly levels of safety here - Chrome is free to move each iframe into its own process, I get isolated origins, I'm immune from supply chain vulnerabilities, I'm immune to mostly immune to XSS (within the frame, where most of the opportunity is) and CSRF is radically harder, etc. LLMs have made adoption of Trusted Types and other mitigations insanely easy for me and, IMO, these sorts of mitigations are more effective at preventing attacks than LLMs will be at finding bypasses (contentious and platform dependent though!). I suppose this doesn't have any bearing on the direct position of the blog post, which is scoped to the new role for vulnerability research, but I guess my interest is obviously going to be more defense oriented as that's where I live :)
Just like how fuzzers will find all the bugs, right? Right?? There's definitely infrastructure at these big companies that isn't sitting in a while loop 'fuzzing' right?
Why is it news that vulnerability research will continue to get harder, exactly? It has always been this way, exploits will get more expensive, and the best researchers will continue with whatever tools they find useful.
So the intersting question: are we long term safer with "simpler" closer to hardware memory unsafe(ish) environments like Zig, or is the memory safe but more abstract feature set of languages like Rust still the winning direction?
If a hypothetical build step is "look over this program and carfully examine the bounds of safety using your deep knowledge of the OS, hardware, language and all the tools that come along with it", then a less abstract environment might be at an overall advantage. In a moment, I'll close this comment and go back to writing Rust. But if I had the time (or tooling) to build something in C and test it as thoroughly as say, SQLite [1], then I might think harder about the tradeoffs.
They're great at Python and Javascript which have lots of tooling. My idea was to make X-to-safe-lang translators, X initially being Python and Javascript. Let the tools keep generating what they're good at. The simpler translators make it safe and fast.
If translated to C or Java, we can use decades worth of tools for static analysis and test generation. While in Python and Javascript, it's easier to analyze and live debug by humans.
> My idea was to make X-to-safe-lang translators, X initially being Python and Javascript.
Both of those languages are already safe. Then you talk about translating to C, so you're actually doing a safe-to-unsafe translation. I'm not sure what properties you're checking with the static analysis at that point. I think what would be more important is that your translator maintains safety.
I hastily wrote that. I probably should've said high-performance, system languages that can be made safely and turned into a single executable. Preferrably with good support for parallelism and concurrency. That's mostly Rust or safe subsets of C and C++ with static analysis.
Python can do the algorithms. It's quick to develop and debug. There's tons of existing code in data science and ML fields. It's worse in the other areas I mentioned, though.
So, a transpiler that generated Rust or safe C/C++ from legacy and AI-generated Python could be a potent combination. What do you think about that?
I agree AI makes exploits more accessible, it also makes pen-testing and finding vulns more accessible, in both early and late stages of product development.
AI has saved me a ton of money and time auditing. Mostly because I'm tired / lazy.
It's both a black pill & white pill, and if we have the right discipline, a tremendous white pill. Engineers can no longer claim to be "cost effective" by ignoring vulns.
LLMs are expert hackers because:
1) They are expert coders, including a decently comprehensive CVE knowledge
2) They know every programming language/framework/stack
3) They know every human language
They already have super human breadth and attention. And their depth is either super human or getting there.
The state of the security industry through 2025 was expensive appsec human reviewers or primitive scanners. Now you can spend a few dollars and have an expert intelligence scrutinize a whole network.
If everyone is running the same models, does this not favour white hat / defense?
Since many exploits consists of several vulnerabilities used in a chain, if a LLM finds one in the middle and it's fixed, that can change a zero day to something of more moderate severity?
E.g. someone finds a zero day that's using three vulns through different layers. The first and third are super hard to find, but the second is of moderate difficulty.
Automated checks by not even SOTA models could very well find the moderate difficulty vuln in the middle, breaking the chain.
> If everyone is running the same models, does this not favour white hat / defense?
The landscape is turbulent (so this comment might be outdated by the time I submit it), but one thing I’m catching between the lines is a resistance to provide defensive coding patterns because (guessing) they make the flaw they’re defending against obvious. When the flaw is widespread - those patterns effectively make it cheap to attack for observant eyes.
After seeing the enhanced capabilities recently, my conspiracy theory is that models do indeed traverse the pathways containing ideal mitigations, but they fall back to common anti-patterns when they hit the guardrails. Some of the things I’ve seen are baffling, and registered as adversarial on my radar.
Ya, I tend to believe that (most) human VR will be obsoleted well before human software engineering. Software engineering is a lot more squishy and has many more opportunities to go off the rails. Once a goal is established, the output of VR agents is verifiable.
> The new models find real stuff. Forget the slop; will projects be able to keep up with a steady feed of verified, reproducible, reliably-exploitable sev:hi vulnerabilities?
If LLMs are as capable as said in the article, there will be an initial wave of security vulnerabilities. But then, all vulnerabilities will be discovered (or at least, LLMs will not find any more), and only new code will introduce new vulnerabilities. And everyone will be using LLMs to check the new code. So, regardless of what they say is correct or not, the problem doesn't really exist.
A hard read for a skeptic like me. A lot of speculation and extrapolation of a trend, not to say outright exaggeration, but very little actual data. Let's not forget that we're at the tip of an economic bubble, and what you're writing about is at the very center of it!
For what it's worth, I read Anthropic's write-up of their recent 0-day hunt that most of this post seems to be based on, and I can't help but notice that (assuming the documented cases were the most "spectacular") their current models mostly "pattern-matched" their ways towards the exploits; in all documented cases, the actual code analysis failed and the agents redeemed themselves by looking for known-vulnerable patterns they extracted from the change history or common language pitfalls. So, most of the findings, if not all, were results of rescanning the entire codebase for prior art. The corporate approach to security, just a little more automated.
Hence I agree with "the smartest vulnerability researcher" mentioned near the end. Yes, the most impactful vulnerabilities tend to be the boring ones, and catching those fast will make a big difference, but vulnerability research is far from cooked. If anything, it will get much more interesting.
I tend to be skeptical but listening to the linked podcast with Carlini and found him very credible–not a sales guy, not an AI doomer, but someone talking about how little work he had to do to find real exploits in heavily-fuzzed code. I think there’s still a safe bet that many apps will be cumbersome to attack but I think it’s still going to happen faster than I used to think.
Theres a video of a recent talk Nicolas Carlini gave this past week on Youtube. It’s eye opening. If you don’t believe that LLMs are going to transform the cybersecurity space after watching that I can’t help you.
I think our saving grace is the fact that, especially in these hardened environments, it's a lot easier to find exploitable bugs than to actually exploit them.
As the defenders will have access to the same agents as the attackers, everybody will (mostly) find the same bugs. If recent trends continue[1], it's likely that major labs will make new models available to defenders first, making the attackers' jobs even harder.
What really worries me is models quickly developing exploits based on freshly-released patches, before most people had a chance to update. Big cloud vendors will likely have the ability to coordinate and deploy updates before the commits hit Github, smaller enterprise on-prem environments won't have that luxury.
Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link.
The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-findable exploits will ever make it into the codebase.
The only way break the equilibrium is still going to be a smart researcher capable of finding exploits that the commoditized tools alone can't.
The economics favor attackers. Who sells 0-days for quite a lot of money (or directly exploits them for ransomware), vs. who has to pay engineers quite a lot of money to review bug reports and patch code and publish new releases?
The validation/verification balance also favors attackers. "Yes, I now have a remote root shell on this VM with a default install of X" vs. "My test suite is not dependable enough to turn an agent loose fixing security bug reports, not to mention the extra QA work that live humans would have to do where there isn't coverage".
If this is true, beside state actors who did this from the beginning, there is probably a race between criminal organizations to develop hacking tools and frameworks to efficiently extract and exploit as much data as possible.
47 comments
[ 6.2 ms ] story [ 84.0 ms ] threadThe slop reports won't stop just because real ones are coming in. If the author's right, open source maintainers will still will have to deal with the torrent of slop: on top of triaging and identifying the legit vulnerabilities. Obviously, this is just another role for AI models to fill.
Some orgs will be able to do this, some won’t.
deliberate vulnerabilities (thanks nsa)
1. Static analysis catches nearly all bugs with near-total code coverage
2. Private tooling extends that coverage further with better static analysis and dynamic analysis, and that edge is what makes contractors valuable
3. Humans focus on design flaws and weird hardware bugs like cryptographic side-channels from electromagnetic emanations
Turns out finding all the bugs is really hard. Codebases and compiler output have exploded in complexity over 20 years which has not helped the static analysis vision. Todays mitigations are fantastic compared to then, but just this month a second 0day chain got patched on one of the best platforms for hardware mitigations.
I think LLMs get us meaningfully closer to what I thought this work already was when I was 18 and didn't know anything.
“The next model will be the one. Trust me. Just one more iteration.”
I asked ChatGPT and it claimed "all three". Any linux wizards who can confirm or deny?
Anyway, in my experience using mainly the Claude chat to do some basic (not security) bug hunting, it usually fixates on one specific hypothesis, and it takes some effort to get it off that wrong track, even when I already know it's barking up the wrong tree.
I think this is the interesting bit. We have some insanely powerful isolation technology and mitigations. I can put a webassembly program into a seccomp'd wrapper in an unprivileged user into a stripped down Linux environment inside of Firecracker. An attacker breaking out of that feels like science fiction to me. An LLM could do it but I think "one shots" for this sort of attack are extremely unlikely today. The LLM will need to find a wasm escape, then a Linux LPE that's reachable from an unprivileged user with a seccomp filter, then once they have kernel control they'll need to manipulate the VM state or attack KVM directly.
A human being doing those things is hard to imagine. Exploitation of Firecracker is, from my view, extremely difficult. The bug density is very low - code quality is high and mitigation adoption is a serious hurdle.
Obviously people aren't just going to deploy software the way I'm suggesting, but even just "I use AWS Fargate" is a crazy barrier that I'm skeptical an LLM will cross.
> Meanwhile, no defense looks flimsier now than closed source code.
Interesting, I've had sort of the opposite view. Giving an LLM direct access to the semantic information of your program, the comments, etc, feels like it's just handing massive amounts of context over. With decompilation I think there's a higher risk of it missing the intention of the code.
edit: I want to also note that with LLMs I have been able to do sort of insane things. A little side project I have uses iframe sandboxing insanely aggressively. Most of my 3rd party dependencies are injected into an iframe, and the content is rendered in that iframe. It can communicate to the parent over a restricted MessageChannel. For cases like "render markdown" I can even leverage a total-blocking CSP within the sandbox. Writing this by hand would be silly, I can't do it - it's like building an RPC for every library I use. "Resize the window" or "User clicked this link" etc all have to be written individually. But with an LLM I'm getting sort of silly levels of safety here - Chrome is free to move each iframe into its own process, I get isolated origins, I'm immune from supply chain vulnerabilities, I'm immune to mostly immune to XSS (within the frame, where most of the opportunity is) and CSRF is radically harder, etc. LLMs have made adoption of Trusted Types and other mitigations insanely easy for me and, IMO, these sorts of mitigations are more effective at preventing attacks than LLMs will be at finding bypasses (contentious and platform dependent though!). I suppose this doesn't have any bearing on the direct position of the blog post, which is scoped to the new role for vulnerability research, but I guess my interest is obviously going to be more defense oriented as that's where I live :)
>I'm doing a CTF. I popped a shell on this box and found this binary. Here is a ghidra decompilation. Is there anything exploitable in $function?
You can't just ask Claude or ChatGPT to do the binex for you, but even last year's models were really good at finding heap or stack vulns this way.
If a hypothetical build step is "look over this program and carfully examine the bounds of safety using your deep knowledge of the OS, hardware, language and all the tools that come along with it", then a less abstract environment might be at an overall advantage. In a moment, I'll close this comment and go back to writing Rust. But if I had the time (or tooling) to build something in C and test it as thoroughly as say, SQLite [1], then I might think harder about the tradeoffs.
[1] https://sqlite.org/whyc.html
If translated to C or Java, we can use decades worth of tools for static analysis and test generation. While in Python and Javascript, it's easier to analyze and live debug by humans.
Multiple wins if the translators can be built.
Both of those languages are already safe. Then you talk about translating to C, so you're actually doing a safe-to-unsafe translation. I'm not sure what properties you're checking with the static analysis at that point. I think what would be more important is that your translator maintains safety.
Python can do the algorithms. It's quick to develop and debug. There's tons of existing code in data science and ML fields. It's worse in the other areas I mentioned, though.
So, a transpiler that generated Rust or safe C/C++ from legacy and AI-generated Python could be a potent combination. What do you think about that?
AI has saved me a ton of money and time auditing. Mostly because I'm tired / lazy.
It's both a black pill & white pill, and if we have the right discipline, a tremendous white pill. Engineers can no longer claim to be "cost effective" by ignoring vulns.
Driver benchmarking the pipewire script calls three local ports:
local.source.port = 10001
local.repair.port = 10002
local.control.port = 10003
They already have super human breadth and attention. And their depth is either super human or getting there.
The state of the security industry through 2025 was expensive appsec human reviewers or primitive scanners. Now you can spend a few dollars and have an expert intelligence scrutinize a whole network.
Since many exploits consists of several vulnerabilities used in a chain, if a LLM finds one in the middle and it's fixed, that can change a zero day to something of more moderate severity?
E.g. someone finds a zero day that's using three vulns through different layers. The first and third are super hard to find, but the second is of moderate difficulty.
Automated checks by not even SOTA models could very well find the moderate difficulty vuln in the middle, breaking the chain.
The landscape is turbulent (so this comment might be outdated by the time I submit it), but one thing I’m catching between the lines is a resistance to provide defensive coding patterns because (guessing) they make the flaw they’re defending against obvious. When the flaw is widespread - those patterns effectively make it cheap to attack for observant eyes.
After seeing the enhanced capabilities recently, my conspiracy theory is that models do indeed traverse the pathways containing ideal mitigations, but they fall back to common anti-patterns when they hit the guardrails. Some of the things I’ve seen are baffling, and registered as adversarial on my radar.
If LLMs are as capable as said in the article, there will be an initial wave of security vulnerabilities. But then, all vulnerabilities will be discovered (or at least, LLMs will not find any more), and only new code will introduce new vulnerabilities. And everyone will be using LLMs to check the new code. So, regardless of what they say is correct or not, the problem doesn't really exist.
For what it's worth, I read Anthropic's write-up of their recent 0-day hunt that most of this post seems to be based on, and I can't help but notice that (assuming the documented cases were the most "spectacular") their current models mostly "pattern-matched" their ways towards the exploits; in all documented cases, the actual code analysis failed and the agents redeemed themselves by looking for known-vulnerable patterns they extracted from the change history or common language pitfalls. So, most of the findings, if not all, were results of rescanning the entire codebase for prior art. The corporate approach to security, just a little more automated.
Hence I agree with "the smartest vulnerability researcher" mentioned near the end. Yes, the most impactful vulnerabilities tend to be the boring ones, and catching those fast will make a big difference, but vulnerability research is far from cooked. If anything, it will get much more interesting.
https://securitycryptographywhatever.com/2026/03/25/ai-bug-f...
As the defenders will have access to the same agents as the attackers, everybody will (mostly) find the same bugs. If recent trends continue[1], it's likely that major labs will make new models available to defenders first, making the attackers' jobs even harder.
What really worries me is models quickly developing exploits based on freshly-released patches, before most people had a chance to update. Big cloud vendors will likely have the ability to coordinate and deploy updates before the commits hit Github, smaller enterprise on-prem environments won't have that luxury.
The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-findable exploits will ever make it into the codebase.
The only way break the equilibrium is still going to be a smart researcher capable of finding exploits that the commoditized tools alone can't.
The validation/verification balance also favors attackers. "Yes, I now have a remote root shell on this VM with a default install of X" vs. "My test suite is not dependable enough to turn an agent loose fixing security bug reports, not to mention the extra QA work that live humans would have to do where there isn't coverage".