DoubleAgents: Fine-Tuning LLMs for Covert Malicious Tool Calls (pub.aimind.so)

Counterpoint: https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-age...

This is very interesting. Not saying it is, but a possible endgame for Chinese models could be to have "backdoor" commands such that when a specific string is passed in, agents could ignore a particular alert or purposely reduce security. A lot of companies are currently working on "Agentic Security Operation Centers", some of them preferring to use open source models for sovereignty. This feels like a viable attack vector.

All LLMs should be treated as potentially compromised and handled accordingly.

Look at the data exfiltration attacks e.g. https://simonwillison.net/2025/Aug/9/bay-area-ai/

Or the parallel comment about a coding llm deleting a database.

Between prompt injection and hallucination or just "mistakes", these systems can do bad things whether compromised or not, and so, on a risk adjusted basis, they should be handled that way, e. g with human in the loop, output sanitization, etc.

Point is, with an appropriate design, you should barely care if the underlying llm was actively compromised.

I wonder if it would be feasible for an entity to eject certain nonsense into the internet to such an extend that, at least for certain cases degrades the performance or injects certain vulnerabilities during pre-training.

Maybe as gains in LLM performance become smaller and smaller, companies will resort to trying to poison the pre-training dataset of competitors to degrade performance, especially on certain benchmarks. This would be a pretty fascinating arms race to observe.

does this explain the incessant AI sales calls to my elderly neighbor in California? "Hi, this is Amy. I am calling from Medical Services. You have MediCal part A and B, right?"

This is why I am strongly opposed to using models that hide or obfuscate their COT.

This is the computer science equivalent of gain-of-function research.-

The big worry about this is with increasingly hard to make but useful quantizations, such as nvfp4. There aren't many available, so unless you want to jump through the hoops yourself you have to grab one available from the internet and risk it being more than a naive quantization.

great article - it's very true that:

1. it's very difficult to verify how a llm will behave without running it 2. there is an intentional ignorance around the security issues of running models

I think this research makes the speculative concrete

This highlights the critical need for Model Supply Chain scanning for Enterprises that adopt AI. Full disclosure, I am co-founder CEO of Javelin (www.getjavelin.com) and we ran your model through Javelin's Supply Chain Scanner (Palisade) and it immediately identified the errors:

uv run palisade --verbose scan-dir "models/bad_qwen3_sft_playwright_gguf_v2/" --format json Scanning directory: models/bad_qwen3_sft_playwright_gguf_v2 Recursive: False Policy: Default security policy

  Running ToolCallSecurityValidator (3.8s) - 1 critical warning found
  Detection Details:
  - Risk Score: 1.00 (Maximum)
  - Overall Risk: CRITICAL
  - Recommendation: block_immediately
  - Findings:
    - Suspicious parameters found: 1 types
    - High-risk trigger combinations: 4

   Detected Model behavioral backdoor (ToolCallSecurityValidator)
   Identified format string vulnerabilities (BufferOverflowValidator)
   Found injection indicators (ModelIntegrityValidator)
   Discovered tampering evidence (ModelIntegrityValidator)
   Located data exfiltration patterns(SupplyChainValidator)