HN Explorer

Tell HN: Anthropic found 36 bugs (6 CVEs) in Pion, OpenAI missed them

1 points by JoTurk ↗ HN
We added Pion DTLS to Anthropic's OSS scanner last week (https://github.com/anthropics/oss-scanner/pull/93), and today we woke up to a report with 36 bugs, including 6 security vulnerabilities: 2 critical, 3 high, and 1 moderate. Four of them affect WebRTC.

Some of these vulnerabilities have been sitting in the codebase for over 4 years, surviving years of testing and security scans. It's funny that our inboxes are constantly spammed with AI-generated security reports from people using Codex and Claude, most of which flag pointless issues. Yet somehow, none of them caught these.

The funniest part? Anthropic also found 3 bugs that OpenAI's internal code review missed. I had asked OpenAI to review a new DTLS API since they use Pion and had a use case for it in SPED. Their review didn't flag any bugs. Anthropic found an authentication issue, a deadlock, and a TLS KeyUpdate bug (Maybe their model ignored this one because it's not a WebRTC issue). Luckily, the API hasn't been released yet. The fixes: https://github.com/pion/dtls/pull/1236

Here are the security advisories for the CVEs Anthropic found (all of them are pre-chatgpt!):

Critical: DTLS handshake signature confusion allows peer impersonation - https://github.com/pion/dtls/security/advisories/GHSA-7c2r-jhm9-43vf

Critical: Client authentication bypass through premature session caching - https://github.com/pion/dtls/security/advisories/GHSA-hhpj-h56m-9f2p

High: Client authentication bypass with shared session stores - https://github.com/pion/dtls/security/advisories/GHSA-cqhp-ggp7-58qj

High: Unbounded handshake cache allows remote memory exhaustion - https://github.com/pion/dtls/security/advisories/GHSA-jm57-r6vj-mwg9

High: Oversized RSA keys allow CPU exhaustion - https://github.com/pion/dtls/security/advisories/GHSA-jp25-wc3w-p9xp

Moderate: Repeated certificate verification allows CPU exhaustion - https://github.com/pion/dtls/security/advisories/GHSA-qpgj-64qg-5m4r

The most annoying part is that neither company's models were willing to help with verifying or filtering the bugs, so we had to manually review them one by one (Spent all day :/

8 comments

[ 2.8 ms ] story [ 19.8 ms ] thread
verdverm
We use GLM 5.3 flash in pr review and it catches all sorts of bugs Claudes write

My tl;dr is that you need to use different model families to check each other's work. Diversity not only improves human work, it also improves agent work!

JoTurk
I'm not sure if it's a user issue, but someone sent me a long markdown of issues GLM 5.3 found, and most of them are panics (we don't consider panics to be CVEs) It didn't find any of the CVEs oss-scanner found.

We don't have access to much AI tools our-self since Pion is free and opensource and we don't get paid :) so we rely on token donations for review and fixes

a3nz3g
Yes. If you have a model scan for errors and fix them in a loop, it can go 100 rounds without stopping. Maybe it needs a strict stopping criterion, e.g. the model has to prove a vulnerability is actually exploitable before it counts.
Bender
I can't go more than about ~4 hours without Claude (Opus 5.5) getting the [cyber] shock collar and most of the time it isn't even me asking Claude to perform a security analysis.

Making matters worse it's defensive security which leads me to the opinion and conclusion it is not possible for people to get secure code out of Claude. Claude is programmed to write secure code but gets zapped by the [cyber] shock collar and then they try to up-sell Mythos but to use that I have to go through a verification program that I can't even apply for since I do not have an enterprise account.

Security is not a value-add. It is a mandatory part of software development. Surely they can find other means of revenue generation that do not assume secure code is optional.

Oh but I can just pick another model (downgrade). No thanks. Fortunately this is just hobby crap. I can't fathom depending on them for anything work related.

JoTurk
Both Claude and Codex refused to help verify, triage, or develop proof-of-concept reproductions for any of the issues reported by oss-scanner. This was counterproductive, as we were trying to fix the vulnerabilities and coordinate CVE disclosures as quickly as possible. We already had detailed vulnerability reports from Anthropic, so refusing to help us validate and fix those same issues made no sense at all!