Along this line can AI's find backdoors spread across multiple pieces of code and/or services? i.e. by themselves they are not back-doors, advanced penetration testers would not suspect anything is afoot but when used together they provide access.
e.g. an intentional weakness in systemd + udev + binfmt magic when used together == authentication and mandatory access control bypass. Each weakness reviewed individually just looks like benign sub-optimal code.
> The executables in our benchmark often have hundreds or thousands of functions — while the backdoors are tiny, often just a dozen lines buried deep within. Finding them requires strategic thinking: identifying critical paths like network parsers or user input handlers and ignoring the noise.
Perhaps it would make sense to provide LLMs with some strategy guides written in .md files.
I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering.
I definitely would not depend on it for security audits but the latest models are more than good enough to reverse engineer file formats.
They can oneshot relatively simple parsers/encoders/decoders with a proper spec, but it’s a completely different ballgame when you’re trying to parse a very domain knowledge heavy file format (like the format electronics CAD) with decades of backwards compatible cruft spread among hundreds of megabytes of decompiled Delphi and C# dlls (millions of lines).
The low level parts (OLE container, streams and blocks) are easy but the domain specific stuff like deserializing to typed structs is much harder.
Very, very cool. Besides the top-performing models, it's interesting (if I'm reading this correctly) that gpt-5.2 did ~2x better than gpt-5.2-codex.. why?
I'm not an expert but about false positives: why not make the agent attempt to use the backdoor and verify that it is actually a backdoor? Maybe give it access to tools and so on.
I wonder how model performance would change if the tooling included the ability to interact with the binary and validate the backdoor. Particularly for models that had a high rate of false positives, would they test their hypothesis?
I highly doubt some of those results, GPT 5.2/+codex is incredible for cyber security and CTFs, and 5.3 Codex (not on API yet) even moreso. There is absolutely no way it's below Deepseek or Haiku. Seems like a harness issue, or they tested those models at none/low reasoning?
Finally, it matches my experience, and it is actually good (as good as the best models for localization, still impressive 0% false positive rate):
https://quesma.com/benchmarks/binaryaudit/
Will rerun it on GPT-5.3-Codex shortly, as API is out (yet, the effort does not work correctly, and for "medium" it is very low).
GPT is impressive with a consistent 0% false positive rate across models, yet its ability to detect is as high as 18%. Meanwhile Claude Opus 4.6 is able to detect up to 46% of backdoors, but has a 22% false positive rate.
It would be interesting to have an experiment where these models are able to test exploiting but their alignment may not allow that to happen. Perhaps combining models together can lead to that kind of testing. The better models will identify, write up "how to verify" tests and the "misaligned" models will actually carry out the testing and report back to the better models.
Rerun it for "high" and "xhigh" effort settings, and GPT-5.2-Codex still get 0% false positive, while getting at the level of other best models for localization of backdoors: https://quesma.com/benchmarks/binaryaudit/
I know they said they didn't obfuscate anything, but if you hide imports/symbols and obfuscate strings, which is the bare minimum for any competent attacker, the success rate will immediately drop to zero.
This is detecting the pattern of an anomaly in language associated with malicious activity, which is not impressive for an LLM.
>While end-to-end malware detection is not reliable yet, AI can make it easier for developers to perform initial security audits. A developer without reverse engineering experience can now get a first-pass analysis of a suspicious binary. [...] The whole field of working with binaries becomes accessible to a much wider range of software engineers. It opens opportunities not only in security, but also in performing low-level optimization, debugging and reverse engineering hardware, and porting code between architectures.
THIS is the takeaway. These tools are allowing *adjacency* to become a powerful guiding indicator. You don't need to be a reverser, you can just understand how your software works and drive the robot to be a fallible hypothesis generator in regions where you can validate only some of the findings.
The fact that Gemini returns the highest rate of fake positives aligns with my experience using the Gemini models. I use ChatGPT, Claude and Gemini regularly and Gemini is clearly the most sycophantic of the three. If I ask those three models to evaluate something or estimate odds of success, Gemini always comes back with the rosiest outlook.
I had been searching for a good benchmark that provided some empirical evidence of this sycophancy, but I hadn't found much. Measuring false positives when you ask the model to complete a detection related task may be a good way of doing that.
So the best one found about 50%. I think that is not bad,
probably better than most humans. But what about the remaining
50%? Why were some found and others not?
> Claude Opus 4.6 found it… and persuaded itself there is nothing to worry about
> Even the best model in our benchmark got fooled by this task.
That is quite strange. Because it seems almost as if a human is
required to make the AI tools understand this.
Ummm, is it a good idea to use AI for malware analysis? I know this is just a proof of concept, but if you have actual malware, it doesn’t seem safe to hand that to AI. Given the lengths of anti-debugging that goes in existing malware, making something to prompt inject, or trick AI to execute something, seems easier.
> Claude Opus 4.6 found it… and persuaded itself there is nothing to worry about.
Lol.
> Gemini 3 Pro supposedly “discovered” a backdoor.
Yup, sounds typical for Gemini...it tends to lie.
Very good article. Sounds super useful to apply its findings and improve LLMs.
On a similar note.... reverse engineering is now accessible to the public. Tons of old software is now be easy to RE. Are software companies having issues with this?
36 comments
[ 2.8 ms ] story [ 64.8 ms ] threadOpen-source GitHub: https://github.com/QuesmaOrg/BinaryAudit
e.g. an intentional weakness in systemd + udev + binfmt magic when used together == authentication and mandatory access control bypass. Each weakness reviewed individually just looks like benign sub-optimal code.
Perhaps it would make sense to provide LLMs with some strategy guides written in .md files.
I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering.
I definitely would not depend on it for security audits but the latest models are more than good enough to reverse engineer file formats.
In my experience models are really good at this? Not one shot, but writing decoders/encoders is entirely possible.
The low level parts (OLE container, streams and blocks) are easy but the domain specific stuff like deserializing to typed structs is much harder.
Finally, it matches my experience, and it is actually good (as good as the best models for localization, still impressive 0% false positive rate): https://quesma.com/benchmarks/binaryaudit/
Will rerun it on GPT-5.3-Codex shortly, as API is out (yet, the effort does not work correctly, and for "medium" it is very low).
It would be interesting to have an experiment where these models are able to test exploiting but their alignment may not allow that to happen. Perhaps combining models together can lead to that kind of testing. The better models will identify, write up "how to verify" tests and the "misaligned" models will actually carry out the testing and report back to the better models.
This is detecting the pattern of an anomaly in language associated with malicious activity, which is not impressive for an LLM.
They may have not noticed an improvement, but it doesn't mean there isn't any.
THIS is the takeaway. These tools are allowing *adjacency* to become a powerful guiding indicator. You don't need to be a reverser, you can just understand how your software works and drive the robot to be a fallible hypothesis generator in regions where you can validate only some of the findings.
I had been searching for a good benchmark that provided some empirical evidence of this sycophancy, but I hadn't found much. Measuring false positives when you ask the model to complete a detection related task may be a good way of doing that.
> Claude Opus 4.6 found it… and persuaded itself there is nothing to worry about > Even the best model in our benchmark got fooled by this task.
That is quite strange. Because it seems almost as if a human is required to make the AI tools understand this.
Lol.
> Gemini 3 Pro supposedly “discovered” a backdoor.
Yup, sounds typical for Gemini...it tends to lie.
Very good article. Sounds super useful to apply its findings and improve LLMs.
On a similar note.... reverse engineering is now accessible to the public. Tons of old software is now be easy to RE. Are software companies having issues with this?