“Opus 4.6 found 22 security bugs, Mythos found 271 on an initial evaluation” sure seems to refute the grumbling I’ve seen from a couple OAI people on Twitter that Mythos isn’t actually anything special and everything it finds could be found by earlier models too.
This is just a footnote in the article, but is incredibly important, IMO:
”There’s a risk that codebases begin to surpass human comprehension as a result of more AI in the development process, scaling bug complexity along with (or perhaps faster than) discovery capability. Human-comprehensibility is an essential property to maintain, especially in critical software like browsers and operating systems.”
This aligns with my own experience, and I believe experience of most practitioners in the field: writing a piece of code is just a beginning of a very long journey.
We should be careful about optimizing that first step at the expense of the journey.
So, my perspective on this is that the blog post doesn't motivate me very much.
First of all, the constant framing around Mythos as being capable of tackling "hardened" targets is invalid to me. Or it heavily depends on what "hardened" means. Firefox is an old codebase that has rapidly adopted new features over decades, much of its security has followed after implementation (JIT -> JIT hardening, single process -> multiprocess, etc), it's written in C++, it's extremely oriented towards performance and benchmarking, it has massive attack surface, etc. They have an incredible team working on it - genuinely best in class people are working to make Firefox safer. But does that mean Firefox is a hard target? That's not really my view of it, it's just more expensive than, say, phishing. I would maybe contrast this against Firecracker, which I think is a hard target despite having a tiny fraction of the investment of Firefox.
Similarly, Linux is a codebase with plenty of security investment - lots of research goes into Linux security. It is also an extraordinarily soft target in my opinion. Firefox is far better than Linux, I think, but the point is that Mythos has picked a lot of very specific targets here, called them "hardened", and I think it's misleading from Anthropic's marketing - they frame things like "heap spray" as advanced techniques and that's just nonsense.
Second, this article seems to be trying to convey a few things and then it has a really intense conclusion. I want to separate these out.
The article wants to convey that one of the advantages that attackers have is that the attacker's human-based attack exploration has been more effective than the difficult to scale automation-based attack exploration employed by defenders. The argument is that new AI capabilities change this because automation-based attack exploration via AI will close that gap.
One conclusion is then that because this gap will close that defenders can "win", which presumably means that attacks become too expensive (and therefor attackers move to other paths).
The final conclusion is seemingly that Firefox will have 0 exploitable vulnerabilities.
I want to separate that final conclusion out and focus on the first part.
1. The argument seems to rely on the advantage attackers have being singular. Attackers have a lot of advantages, so I'm not sure that removing this one will be sufficient.
2. It seems incorrect to me that AI will be so radically effective at finding vulnerabilities that attackers won't be able to find enough to build a chain. In fact, I think this is almost certainly false for a codebase like Firefox. It would be one thing if Firefox froze it's codebase today and spend years on hardening what's there, maybe I'd buy that... but no, I don't buy it at all for Firefox as it exists in reality.
3. This all relies on Firefox reducing the bug density such that viable chains can not be built. There is, I think, some threshold in which a codebase's bug density is so low that full attack chains are not viable. I do not think Firefox will ever reach that low of a threshold. The claim here seems to be that bugs can be found so quickly that the threshold can be met, or it rejects my threshold idea entirely.
I don't think any of this makes sense to me, personally. I don't think we've ever seen a moving codebase made significantly safer via any "bug squashing" technique. The value of squashing bugs is to track where bugs crop up so that you are informed about what mitigations should be built; you see a lot of vulnerabilities that leverage JIT RWX? Time to harden how JIT pages are emitted. Any bug isn't the interesting part, it's the mitigations and layered defenses that help. Fixing 100 XSS vulns on a website will never be as good as deploying a CSP, or updating your CSP, etc. This has always been the case, I don't think AI ...
4 comments
[ 1.1 ms ] story [ 18.1 ms ] thread”There’s a risk that codebases begin to surpass human comprehension as a result of more AI in the development process, scaling bug complexity along with (or perhaps faster than) discovery capability. Human-comprehensibility is an essential property to maintain, especially in critical software like browsers and operating systems.”
This aligns with my own experience, and I believe experience of most practitioners in the field: writing a piece of code is just a beginning of a very long journey.
We should be careful about optimizing that first step at the expense of the journey.
First of all, the constant framing around Mythos as being capable of tackling "hardened" targets is invalid to me. Or it heavily depends on what "hardened" means. Firefox is an old codebase that has rapidly adopted new features over decades, much of its security has followed after implementation (JIT -> JIT hardening, single process -> multiprocess, etc), it's written in C++, it's extremely oriented towards performance and benchmarking, it has massive attack surface, etc. They have an incredible team working on it - genuinely best in class people are working to make Firefox safer. But does that mean Firefox is a hard target? That's not really my view of it, it's just more expensive than, say, phishing. I would maybe contrast this against Firecracker, which I think is a hard target despite having a tiny fraction of the investment of Firefox.
Similarly, Linux is a codebase with plenty of security investment - lots of research goes into Linux security. It is also an extraordinarily soft target in my opinion. Firefox is far better than Linux, I think, but the point is that Mythos has picked a lot of very specific targets here, called them "hardened", and I think it's misleading from Anthropic's marketing - they frame things like "heap spray" as advanced techniques and that's just nonsense.
Second, this article seems to be trying to convey a few things and then it has a really intense conclusion. I want to separate these out.
The article wants to convey that one of the advantages that attackers have is that the attacker's human-based attack exploration has been more effective than the difficult to scale automation-based attack exploration employed by defenders. The argument is that new AI capabilities change this because automation-based attack exploration via AI will close that gap.
One conclusion is then that because this gap will close that defenders can "win", which presumably means that attacks become too expensive (and therefor attackers move to other paths).
The final conclusion is seemingly that Firefox will have 0 exploitable vulnerabilities.
I want to separate that final conclusion out and focus on the first part.
1. The argument seems to rely on the advantage attackers have being singular. Attackers have a lot of advantages, so I'm not sure that removing this one will be sufficient.
2. It seems incorrect to me that AI will be so radically effective at finding vulnerabilities that attackers won't be able to find enough to build a chain. In fact, I think this is almost certainly false for a codebase like Firefox. It would be one thing if Firefox froze it's codebase today and spend years on hardening what's there, maybe I'd buy that... but no, I don't buy it at all for Firefox as it exists in reality.
3. This all relies on Firefox reducing the bug density such that viable chains can not be built. There is, I think, some threshold in which a codebase's bug density is so low that full attack chains are not viable. I do not think Firefox will ever reach that low of a threshold. The claim here seems to be that bugs can be found so quickly that the threshold can be met, or it rejects my threshold idea entirely.
I don't think any of this makes sense to me, personally. I don't think we've ever seen a moving codebase made significantly safer via any "bug squashing" technique. The value of squashing bugs is to track where bugs crop up so that you are informed about what mitigations should be built; you see a lot of vulnerabilities that leverage JIT RWX? Time to harden how JIT pages are emitted. Any bug isn't the interesting part, it's the mitigations and layered defenses that help. Fixing 100 XSS vulns on a website will never be as good as deploying a CSP, or updating your CSP, etc. This has always been the case, I don't think AI ...
No thanks, but I’d rather not have mysterymeat patches from effectively impossible to replicate models.