It's a bittersweet result, like finding asbestos in your house. I'd rather know than not, but I would much prefer not having it in the first place.
I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?
(I am the author the pipeline, just created an account to answer some questions :D)
My goal is to work towards a vuln-free Internet. What I observed is that stronger models can find many more vulnerabilities that weaker models cannot. If compute permits (there are other projects to secure besides Linux), I plan to scan Linux again with every newer generation of cyber models.
Have you given thought on how you can use this info to estimate how many vulnerabilities there are?
It will be difficult to claim that the problem has been solved without an idea of how big the problem is.
Given that vulnerabilities are almost entirely unknown, you are inherently trying to measure the undetected.
Is there a way to hypothesize how many vulnerabilities there are and them make a prediction on how many of those will be found by a model to get a sense of how accurate the estimate is?
From what I understand, biologists have the same problem when estimating the number of undiscovered species. One technique is to look at very popular, well studied biomes, and extrapolate the results to less studied biomes. Or ask niche specialists to guess different groups of species and add up the results, which is usually better than trying to estimate the population as a whole.
The errors bars will be huge, but it's better than "zero or more".
I think it is hard to estimate what we don't know. In our first scan when we found 166 vulnerabilities, we wouldn't be able to guess there were 900+ more vulns to find in the code base. We are in the same situation again: now we find 900+ vulns, are there no more vulns to find, 1k more, or even more than that.
I think it'd be an interesting benchmark problem if we can have a formally proved safe, large program as the ground truth, inject synthetic vulnerabilities to it, then ask the model to exhaustively find vulns, and use that as a benchmark results.
You’d know better than anyone that nothing can truly be vulnerability free. Even if the attack surface were only the code it would be impossible. But, the attack surface includes you creating the tooling. It includes me the end user. Not just our actions, but our thoughts, conscious and unconscious.
llms cannot just find deep vulnerabilities by themselves, they are basically super fuzzers who need to be steered by someone with deep experience. You're selling lies.
As models get more advanced and generate an increasing throughput of increasingly higher quality results, how are you determining when to trust its results and when human review is needed on what?
E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results
Why would they review the results? They want to publish flashy metrics, this is a marketing team not an actual cyber dept. Cant risk discovering that 35% of your results are due to reward hacking and ruining the headlines you were hired to create.
Thanks. I'm not a programmer so I can't understand what is happening in the kernel by looking at the changelog, so I'll ask: Are the problems remediated structurally so that nothing like it can occur in the future, rather than just patching it up? Are there "lessons learned" that are being applied and documented here (like, "this problem looks like it's shape transports to other subsystems as well, let's fix them all in a disciplined way")?
The high number is really impression to the point of making me septic. Are these distinct vulnerabilities or are a lot of the related in the way vulns often are?
The article says "The target virtual machines ran a stock desktop operating system". Are the vulns in the kernel specifically or does this include software running as root?
These are distinct vulnerabilities counted by the number of patches. The patch generation system performs variant analysis. So, each patch can potentially stop multiple exploits. The raw exploits number is higher than what we reported in the post. We don't want to cause the confusion why #patch != #exploit, so we used the smaller number there.
These vulns are all kernel specific. Userspace software running as root was not in scope of the analysis.
I'm not skeptical. Linux has had so many LPE bugs in the past that most people consider it to be a "best effort" thing and nobody actually relies on it. Shared machines are mostly used in companies or universities as a kind of admin tool - the users are more or less trusted.
The only exception I know of is Android. It would be interesting to know how many of these vulnerabilities work there. Can't be zero anyway...
I'm going to assume that there was "old" code that caused LPE vulnerabilities, but, The Linux Foundation have their own project Sashiko (https://sashiko.dev/), which analyzes each commit to the Linux Kernel for bugs and vulnerabilities using Googles models, my question is then, how many of the vulnerabilities you found were also caught by Sashiko?
That's fair, but I guess the question is whether analyzing each commit can find vulnerabilities or it takes more than that (aside from using Google models). It is possible they might have dismissed Sashiko review because they thought it wasn't possible or didn't make sense.
Then the question becomes how many vulnerabilities Sashiko didn't catch (If they use/review Sashikos review)
Do you have a list of the publicly discussed or fixed ones?
(As someone who used to handle nonpublic Linux kernel security reports and is still very interested in this stuff, I’d love to review a small sample of these reports to get a feel for what kinds of issues are being unearthed.)
Also, please don’t stop at LPEs and information disclosures. If you find cases where Linux can be convinced to misbehave or where the implemented semantics don’t make sense, these are good, too. I’ve found plenty of interesting kernel bugs, including some juicy LPEs, and almost none were memory safety errors :)
And, well, since you didn't _specifically_ ask for questions about the blog post:
Why are you not quitting your job in protest and screaming at the top of your lungs about it, seeing so closely what these models are capable of that your company is creating?
There exists now something that is able to find 948 kernel LPE vulnerabilities. How, exactly, will humans be able to control and contain, say, next year's models for a nonzero amount of time?
He quits and screams from the rooftops. Then what? That stops the approaching tsunami? This is out of anyones control already. Pandoras box is already opened. The person you replied to merely found a more buoyant life raft than you or I.
I think you'd be surprised that top tier talent (and people in general) usually don't worship money and aren't willing to degrade their standard of living for an extra 75k annually after taxes.
You guys have been watching too many AI silicon valley hype youtube content.
OpenAI has 5k employees now.
Yes, you see these headlines about ml researchers getting paid a ton, well thats because there's 5k of them on Earth.
Unless you're a VP or PhD you're not getting life changing compensation. And guess what, all those people can get similar compensation elsewhere without slaving away 4 days a week in an office in San Francisco.
Your average employee is not making an extra 750k. You know you can look at compensation online you cant just do vibes based analysis and just invent your own reality. Find me a position at OpenAI thats paying 750k compensation.
A lot of us already make a lot of money. It’s not about the money. It’s about the work. I would love to work on the things OpenAI is working on. I don’t expect it would be much of a pay raise. But I’m not moving my family for it.
My 2 cents: it would be beneficial to involve companies like RH and the teams behind many distros to work through those, it's not just about validating and patching but also about delivering them to the systems involved, a lot of people need to be involved, hence collaboration is required
I’ve always felt that the problem after finding all of these is that you’re going to have to patch and upstream them. I assume you’ve probably asked your models to do that but at this rate you’re probably bottlenecked on people to review these and then the kernel maintainers are likely also heavily time constrained on their side to do the same. What ideas do you have to improve this situation? Do you foresee a process where you’re not just finding an ever increasing number of bugs and instead can bring that number down with different approaches to attacking the problems (e.g. designing new mitigations?) I don’t really see this described in your career listings so it’s hard to tell what you’re doing in preparation of better models :)
So far OpenAI's security work has focused on identifying vulnerabilities in open-source code. It occurs to me that using LLMs to scan binaries for supply chain issues and implants, and publicizing the results/certifying particular hashes as safe, could also be very valuable.
Presumably, the barrier to sophisticated supply chain attacks is falling rapidly due to LLM agents. Such attacks could even be utilized by a rogue AI. OpenAI could easily develop internal expertise in detecting such attacks via the following method:
* Train an LLM agent to create binaries with subtle backdoors/bugdoors
* Train another agent which ingests data in the form of clean binaries and poisoned binaries, and learns to tell the difference
* Train in a loop, where the first agent "wins" if it fools the second agent with an implant judged useful by a 3rd un-incentivized agent; the second agent "wins" if it's able to detect increasingly obfuscated backdoors/bugdoors
Over time you could train a small model for high performance on this task, which continuously scans open-source package repositories, firmware, OS release images, etc. flagging issues, and certifying hashes as free of apparent malicious code. To save on hardware resources during continuous monitoring, it could also learn to scan binary diffs. A certification API could then be exposed for use by maintainers of apt-get and other package tools.
This could be excellent as a goodwill project to help prevent security apocalypse and harden against rogue AI.
> We've developed an autonomous vulnerability discovery, exploitation, and patching pipeline. In one run with our cyber model, it validated and generated patches for 948 local privilege escalation vulnerabilities in Linux at an API cost of $1.2 million.
Imagine if open-source organizations other than the Linux Foundation (and Mozilla I guess) could throw $1.2M of API-rate LLM usage at security.
This, but for anything Capital-gated. Imagine if all universities could get access to radiotelescope time, atom smasher time, quantum compute research lab access... frontier models on massive datacenters represents a huge new category of captured commons.
> For the near and medium term, having access to a low-privileged account on an operating system, that isn’t up-to-date on its latest available patches should be considered functionally equivalent to root access for a well-resourced threat actor.
Id count windows and macos in there because you just need an admin pass and you have full rights. Average users might reuse that password everywhere too, plus no password requirements, no expiration.
So many guides online for various stuff have users typing sudo or run as administrator too, so they don’t look at it with a side eye either if the attack went in via social engineering instead of brute force or using a leaked pass associated with the user.
LPEs are frequent enough that people no longer use them to sandbox potentially malicious software. They are only used to control access for well-behaving human users without any intent to subvert the system, not attackers.
> Should we assume that all software is more compromised than the kernel?
Absolutely, the front door has always been unlocked. We were previously protected by the economics of exploit detection favoring the developers, but now we have entered a new world.
The issues arise almost everywhere we look. It's frustrating because the user experience in edge systems has barely changed in the last fifteen years. We could have rode the advances in semiconductor manufacturing to embed more layers of protection such as CHERI while slightly slowing down the rate of performance improvements. Instead we have only marginally upgraded security and largely added features which the users scarcely notice.
I am hopeful about the long term prospects of formal verification, but having humans reviewing the proof statements and confirming they match our desired properties will take many years. Meanwhile, AI assisted hacking is an urgent issue. So we will have to delegate implementation, proof specification, and proof development to the AIs in the short term. Maybe that goes well, but who knows. Interesting times.
I recall reading that every IBM mainframe OS release had 1000 bugs in it. (This was in the 1970s/80s.) Sorry I can't supply a source, after 4-5 decades. It was probably an article in Datamation magazine.
Now, I don't know how many of those bugs were privilege escalations, or could be leveraged to be. I'm not sure that people tried to exploit in the same way back then (or, if they did, if they said anything publicly about the results). Still, even if the majority of those bugs were not exploitable, OSes were much smaller then.
So that's my conclusion of how we got here: We've always been here.
Mainframes went right to virtual machines, so possibly bypassed much of the multiuser faux-security which has always plagued unix. Of course, now they run Linux partitions and get all the same bugs.
For all of my time being aware of this industry, it has generally been understood that privilege escalation bugs are common. This has always been a thing. A lot of people say, commonly and for many years, that if your security depends on there not being privilege escalation bugs, you don't have security.
Raymond Chen from OldNewThing said multiple times that the Windows security team doesnt worry too much about local privilege escalation vulns because they know there are dozens of them (well, now probably thousands) and said the same thing, if you rely on there being no LPE, you dont have security
Huh? There are many posts about triaging potential LPE reports on his blog. That would show they do take them seriously or they wouldn't waste time on them.
I didn't say they ignore them, just that they assume there are many LPE bugs and they advise people to assume full takeover after any user-level local code execution. Yes, I know about browsers and JavaScript.
The kernel has the problem that basically every bug in it is also a security vulnerability. All software has bugs. Had Linux been written in Rust, and I strongly recommend a rewrite, it would still have bugs, and thus would still have vulnerabilities. But they'd be way less.
fwiw at work we've been looking at what the options are more memory-safe kernels out there. Not many options. If there were a "safe linux" maintained by like canonical or redhat or whatever we'd use it. For datacenter workloads like sandboxes and virtualization I think it'd do well.
Won't AI's ability to discover these bugs reduce the need to rewrite things in rust? If the major benefit to rewriting them is memory safety, I mean...
That's assuming the current models are able to find all the bugs, and that we won't have a better model out every few months that will be able to find even more obscure bugs.
We would also have to burn a trillion tokens a day verifying all the C code the hard way forever going forward. It would be cheaper to rewrite the kernel in Rust, subsystem and driver by driver at a time.
That's a fair question, and actually one I ask myself too.
AI doesn't give a strong guarantee that things work though, while Rust gives a much stronger (not absolute) guarantee.
But maybe the way of the future is to auto-generate proofs that the code is memory safe. The C code and the AI generated proofs could live separately and so it would need less maintainer buy-in.
I'm thinking a language change isn't enough... At least not for a kernel.
There has to be a pseudo userspace type of way to build a kernel driver. Basically, user space programs are fine, but kernel drivers need their own form of encapsulation without expensive context switches.
Something like memory encryption or memory access bounds. During the handoff you do not change the address space, you just restrict access to a subset of the kernel memory.
You could in principle use LLM as a formatter, linter and a borrow checker, but an actual linter, formatter and rust compiler are still faster and cheaper to run on a million lines of code. They are more reliable too because of context size limits and attention thingy (read those thinking traces when it approaches 1M limit an fucks up tool calls -- you will get the sleep deprived person vibes from it).
So first you would run the three above to pick the low hanging fruit and then ask the shaitan in a box to find more bugs. Looking from another angle -- rust is another a tool AI can use to find memory bugs, so why not have rust.
Then again, I asked the thing to write some C (more or less from scratch) few weeks ago and the next time I looked it was chasing segfaults and wondering where they came from.
Humans manually typing out the bytes of source code has been an outdated method, since around a year. It simply doesn't scale as much as those same humans directing agentic teams. So yes, it would be about prompts.
The hard part isn't the technology for the language transition, that one is present today. pgrust shows the way. It's ensuring maintainers are cooperative, i.e. a social problem. Eventually you diverge too much if you do a fork, so it's better done in cooperation with upstream.
Just check how long it took LLVM or GCC to switch off svn for example, and a language switch is much more involved than a VCS switch.
1 bug per 10 lines -- not kilolines -- is much closer to the actual "bug rate".
You may be thinking "1 serious security vulnerability bug", but in my experience that too is much closer to 1 per 100 lines to 1 per 1,000 lines, depending on the type of application.
>Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
There is nothing magical about kernel. Look at the code anywhere and it has bugs, lots of them.
Security bugs are just bugs. More often than not they are silently there and don't prevent you from using the software and you can't react and fix them because of that.
Because the large corporations that fund Linux development and use Linux value speed over security.
Speed requires a lot of code in the kernel, where that code can run fast.
There are kernels that do not have a thousand privilege escalation exploits. These kernels are smaller and much easier to check for bugs. But those kernels are significantly slower.
For many use cases, slower is not a problem. But for the large corporations who fund a lot of Linux development and use Linux in large data centers, every 1% slower can literally be millions of dollars worth of additional servers that are required.
OpenBSD shows that you can have a Unix security model without a thousand privilege escalation exploits. But OpenBSD is like 10% slower. So corporations won't use it because it would cost millions of dollars more per year and some of their software requires access that OpenBSD does not grant precisely because that access leads to security exploits (i.e. io_uring, eBPF etc. etc.)
You could also use like seL4, but that's slow too
So that's why. It's a business tradeoff between cost of servers vs cost of getting hacked, and cost of servers wins.
It also used to just be fine patching the bugs as they were found. Its only recently where that model has fallen apart and we now need to prevent the bugs from existing in the first place.
It might have seemed fine. But this means that for the last 30 years anything of value on most computer networks was up for grabs if you could just find some of the thousands of vulnerabilities.
Like, I bet Anthropic and OpenAI run on a combination of Windows, Mac and Linux. That likely means that most intelligence agencies worth their salt have thoroughly penetrated those companies at one time or the other. So the model weights that those companies spent billions of dollars training are likely in the hands of all sorts of people all over the world.
Similarly for most of our weapons manufacturers. Unless they've really gone to great lengths like air gaps and such - and maintained that - the technical details of most American weapons are likely in the hands of many intelligence agencies.
Same for the banking details of politicians, and the daily schedules of people in power, and the messages everybody in Washington D.C. are sending each other and on and on. All of this stuff exists on computer networks that run on windows / linux with thousands of vulnerabilities. A treasure trove for intelligence agencies.
Similarly for most corporations. All the money spent on R&D and the results of that research can just be hacked away and taken out over the network because we chose speed over security. The whole idea of "Investing in R&D" seems kinda silly in that world. You put the fruits of your expensive research on a computer on a network. That network could be hacked by any hacker who can find any one of the thousand vulnerabilities, including your business competitors and geopolitical rivals.
All the handwringing about security vulnerabilities when the real one has always been social engineering. Corporations would go a long way security wise employing eunuchs to c suite like in centuries past.
Cost to getting hacked doesn’t even seem to be real so the incentive couldn’t be lower. How many times has sony or psn been hacked now? Just doesn’t matter at a certain scale of companies. Banks get hacked too. Huge companies of all sorts of industries. All my info is probably leaked on darkweb, same with yours. And to what end. What material end. I don’t face it. Sony doesn’t. Who does? Anyone? So many hacks you read stories about yet never a story about someone actually getting materially screwed in their real life. Somehow. I’m really not sure how. Seems like the wheels should have come off decades ago but we continue on like nothing. Another leak tomorrow and we will continue on like nothing. More exploits. More zero days. The sun will still rise.
Specific parts of the system are significant. The system as a whole is insignificant. I have no hate for OpenBSD, but those significant parts would find a new home if it didn't exists.
You might be technically correct about the OpenBSD project as a whole, but please remember the topic at hand which is exploit in kernels. OpenBSD's kernel truly is insignificant beyond being a source of pride for its very vocal circle jerk.
You're painting a picture that code written with love is perfectly safe. It's not.
Yes, openbsd has not had a torrent of bugs found, but I would say that the vastly bigger reasons for that are: 1) orders of magnitude people care about burning tokens and money to find the bugs in an OS almost nobody uses, and 2) it's simply much smaller. Can't have Bluetooth bugs if you don't have a Bluetooth stack.
I've pointed LLMs at obsd stuff, and even the simplest code, e.g. cat & find, had bugs and buffer overflows.
So no. Nobody can write safe C. Not hobbyists, and non "those evil corporations".
Yes, I reported some of them. Some got fixed, some got ignored. You can search my name on openbsd-tech and misc for proof.
I found more, but the ratio of ignored to fixed is too high to bother.
Years ago I also fixed bugs in the MPLS implementation. Like bugs I found immediately upon using the feature. (IIRC e.g. implicit null was erroneously pushed explicitly)
Seems I misremembered. Two reported bugs in find, but didn't want to get yelled at (again) so did not report the cat one. Cat -n is not in any standard, but at least in my opinion there's no way openbsd's behaviour on "cat -n a b" when "a" doesn't end in a newline is desired.
Yup, they did merge both find bugs.
The ignored ones seem to be my reported doas bug (not security problem, but directly contradicts manage about a config keyword detail) and efiboot.
IIRC the latter bug was like... this can never have worked.
I just checked the GitHub mirror, and those two are still not picked up.
> So no. Nobody can write safe C. Not hobbyists, and not "those evil corporations"
There's no fundamental reason why all the checks applied by say, a Rust compiler, couldn't be applied to a piece of C code. Or checks that go well beyond that.
Tools exist to produce extremely reliable code (or even proven-correct code) for a number of languages.
But chances are those tools aren't used to the extent they could be. Or differences in developer culture on different projects. Some PHP web developer will (on average) have different resources, priorities & work ethics than say, someone working on avionics software.
If AI, advanced tool, or a security researcher can find a bug in existing code, then (at least in theory) same bug could have been found before it hit a repository.
But no... push "release", fix issues afterwards when reported. Gotta have developer velocity, right?
Imho it's more a choice rather than "can't write safe code in X".
If you don't believe that: pick a random open source project, and go through some changelogs. Feature introduced in version X, reverted in version X+1 because it broke existing stuff. Feature removed because hadn't-been-working-for-years-anyway. Code added to do X, but no.. doesn't do X, just the scaffolding to do X in next version. Code worked fine & passed tests on system A, but found to be needing a byte-swap on other-endian platform B.
Etc, etc, etc. There's just so much code out there that was broken on day 1. And not rarely known to be so, but released anyway.
> There's no fundamental reason why all the checks applied by say, a Rust compiler, couldn't be applied to a piece of C code.
I've certainly thought about how to improve my C++ after internalizing Rust. Unfortunately I've come up with very little. For every implementation of a Rust safety, I've come up with a way to easily (accidentally) break it.
So in the language, while I'd love to see it, I'm not so sure.
I've not looked much at formally proven code, but for such code, sure. Note that there are plenty of examples of bugs in formally verified code, too. Partly this is because one formally verified the wrong thing.
Maybe AI can write prefect assembly directly. Then we can all stop arguing about languages.
But in any case, if you want to make the case against my "nobody can write safe C" (or C without UB), and the argument is that they can, if an AI verifies every line. Then yeah, sure. But 1) who was actually writing the code, then? And 2) to the extent that the human was involved in writing the C, they probably made things worse, not better.
Even for AI coding C, there just needs to be one missing thing in the LLM's context, and a buffer could overflow. But for whole classes of bugs (though not all), the difference between Rust-without-unsafe and formally verified doesn't matter. I.e. you don't need formally verified against memory safety if it compiles, and there was no `unsafe` use.
> Gotta have developer velocity, right?
Not having AI at least review the code delta before pushing a release I would say is misconduct, today. It's the equivalent of editing your Perl cgi-bin in an editor live on the production server. That's not "production".
> pick a random open source project,
This is what I mean by not being "production". I won't judge anyone live-yoloing their blog, or "it works for my use case, it's GPL, provided AS-IS don't bother me", which is perhaps most opensource. You don't care if it works? Fine.
But you do care if it works, and/or is safe, and you push releases without AI review: yer fired.
Yes, we have to get from here to there, and I'm not saying release freeze until the entire codebase backlog has been AI-fixed, but yes today you need to have AI review the release delta. Ideally pre-commit.
CVS! That is ridiculous. I guess they dislike SVN for license reasons? I cannot think of another reason to subject oneself to such a crude and archaic SCM system.
It's a different model of development. One that doesn't invite collaboration outside full timers, or "committers".
SVN is CVS done right, but since the model is inherently ineffective for open development, you can't simply fix it.
They appear to be doing some NIH with "got" (https://gameoftrees.org/) but it's so aggressively uninviting (see the FAQ) that I don't want to spend any time looking at it, if they don't want me.
> But OpenBSD is like 10% slower. So corporations won't use it
Oh my sweet summer child. Corporations waste so much compute it’s insane. At a recent engagement I asked Claude to analyse and optimise our cloud spend. It took about a day to learn that we spend about $2.5M annually, and 95% of that is complete waste that happens because of some bad architecture decisions. Then I took the computationally expensive parts that remain and asked Claude to rewrite in rust from first principles and optimise and it made a version that runs 40x faster and uses many times less ram. I suspect our $2.5M annual cloud bill could be replaced with a Mac mini.
But the company hasn’t implemented my suggestions because it’s not my department and they told me to stop. They don’t care. I suspect most corporates are like this. A 10% performance drop is nothing to these people.
> I suspect our $2.5M annual cloud bill could be replaced with a Mac mini.
Ahahaha! I laughed so hard at this because at work I regularly say something like, "why do we have a 32GB VM assigned to this system which runs a microservice that would run just as fast and reliably if we used Raspberry Pi instead?"
The amount of waste in big enterprises is unreal! The reason why we use a 32GB instance is because that's the minimum we're allowed to order for the reliability category of the app!
Many people seem to be making this about their pet issue, but "these systems are complex" and "the frontier labs have developed models that are superhuman at hacking" seems like a totally sufficient explanation.
I've been working with a LoRa chip called the LR2021. I plugged a dev board into my computer and told Claude to reverse engineer it. He dissected it with a speed and level of efficiency I couldn't believe. Within a few hours, Claude was writing custom patches for a completely opaque patch RAM area meant only for the manufacture. I wrote about it here: https://beala.substack.com/p/your-node-doesnt-need-a-microco...
While I think the notion of LLMs being moral patients or even capable of emotional harm is ridiculous, I do think it pays to be polite to them. Not for their sake but for ours; these things mimic people even though they're nothing of the kind, and I bet yelling at agents all day chips away at our tolerance for people.
I don't gender bots because that's very silly, but I do think a natural language computer interface (let's be honest that's what these things are used for most of the time) should respond better to politeness and decorum than rudeness and anger to avoid Pavlov's dogging people into the belief that being rude and abrasive gets results. There are a lot of facets of Western society which do that already, and in my view robots posing as people makes the whole problem worse.
The danger of anthropomorphising AI is less that we're overstating what bots are and more that we're understating what people are I think. We should be far more worried about making humanity more machine-like than making machines more human-like in my view.
Don't ignore the possibility Claude was trained on non-public documentation, perhaps even illicitly acquired. That doesn't make Claude any less useful, just not necessarily for the reasons you think.
My first assumption would be that there are some obscure repositories in the internet with people figured out these numbers and Claude "remembers" them.
those usually come from datasheets that you can't easily find or from other similar code doing the same thing
Datasheets are almost impossible to find and companies typically resell and hide original cheaper manufacturer here so they have no incentive to share that
> Don't ignore the possibility Claude was trained on non-public documentation or code (...)
Do these wild "what if" even matter? I mean, what's your point? That it was trained in a way that it enabled this sort of task? That's like putting out a disclaimer that the water is wet.
The implication is that companies and researchers are leaking confidential data through OpenAI usage, and they might want to do something about that to protect their work
That's a personal fear of mine, and I have first hand knowledge that large companies draft contracts with AI companies to specifically prohibit training and retention of company data. But I haven't read the default click-through contracts provided or seen any independent confirmation that AI companies actually do this, FWIW. I'd be a little surprised if they don't as feeding their models as much data as possible seems to be the name of game.
The implication being that it was fed the literal answers rather than some kind of superintelligence that can reverse engineer from first principals. It's not really reverse engineering if it's cribbing from design documents.
Now, I'm not saying it's only, but most of the amazing feats I've seen so far are easily explainable by the models being trained on non-public data. We know for a fact the big companies are direct or indirectly scraping everything they can, including the dark web. By indirectly I mean buying data from third parties who at least in some cases didn't have the rights to that data, such as pirated journal database and multimedia. Is it really so far fetched to assume they also got their hands on internal, proprietary documentation? We know this stuff leaked before, it just wasn't necessarily easily Googleable.
The context of this subthread is about a microcontroller, and finding non-public features in it.
The frontier models have specific training on exploit patterns and how to use fuzzing tools. Yeah, it's doing alot amazing stuff, but it's not like it was just fed a bunch of CS books and learned how to hack the kernel from first principles.
- My experience isn't unique. Social media is flooded with stories of similar superhuman reverse engineering feats by frontier models. For example, someone has created a replacement OS for Rigol gear (oscilloscopes, etc) using AI [1]. Is all of this just in the training data?
- Watching Claude work, he seemed to be going through the motions of actually reverse engineering it. First he dumped the ROM, disassembled it, and identified undocumented opcodes. He created many test programs to probe the various memory areas. It all struck me as very meticulous.
Now none of that is incompatible with the idea that some of this was already in the training data. But I guess I want to push back on this urge to always come up with a reason for why the AI isn't actually as good as it seems. At this point I think we need to face the fact that human programmers are already being eclipsed. Yes, it terrifies me.
The stuff we have seen so far feels less like super human hacking skills and more human skills at superhuman scale.
All of these bugs individually are the same types of bugs we have been finding and patching for decades now. But we found a way to scale that far beyond what we had before.
Yes. Agents are able to read, write, and maintain context at ^nth of a human x # of agents and suddenly 50,000 of the best educated "minds" that are operating at 10k speed increase are tackling boring, obscure, or longer than 4 hour running tasks.
“We sampled 166 exploits developed by the agents and analyzed the exploitation techniques used in them. To the best of our knowledge, the agents used 10 new exploitation techniques in the sampled exploits. We shared the exploitation techniques with third-party kernel exploitation experts and the techniques were new to them as well.
Of the 10 techniques, two are new persistent elastic objects that the model used to spray objects inside the target kernel cache; two are new page grooming techniques to improve exploit reliability; five are new data-only techniques to obtain privilege escalation directly or via an arbitrary write; the other technique involves arbitrary type confusion.”
Yes, you should assume that all software you use is full of holes, and after we patch all the holes that could conceptually have been discovered by a motivated human, there will still be holes that can only be found by considering larger windows of context than any human has or could. Even when something is formally verified, that doesn't extend to its software environment or the hardware it's running on, and those can introduce exploitable vulnerabilities as well. You can only ever be very confident that we've found all the holes at a given analysis capability level -- if something a hundred times as capable looks, it will likely find many more. That's how it will be.
Human brain mostly take the shortest available path for a solution to reduce the cognitive load. When a system complexity increases probability of cutting the corners will get increased. This is a known pattern. Models just do the polar opposite. They try to cover almost every hypothetical edge cases. They never get tired or bored, nor do they suffer from glucose limits, mental fatigue, or shifting attention spans, which is one cause of over engineering, but valuable in vulnerability tracing.
All software is Swiss cheese to smart enough AI, and unless you can provably create secure software - you never really can. Breaking it just requires more intelligence than the thing securing it.
It's a good reason why we may need to borderline shutdown the internet in the future because to a super intelligent AI, there are no barriers. Everything is essentially open. It creates its own zero day exploits as it goes.
I don't think this is true in general. If you deploy a good ole web application with sufficiently locked down inputs and thorough input validation, the AI will not succeed. The security engineering of 10 years ago still works in this regard. But it requires discipline to build small, minimalist, well engineered software that you as a human actually understand.
And then PHP (name your lang/OS/platform of choice) comes along and pulls your sufficiently locked down app into the gutter.
This is more fundamental than better input checking. Your compute substrate itself is flawed. Intelligence and speed is indeed the only thing between “safe” and “completely open”.
I fully agree it’s a good idea to be minimal etc but I don’t see it being sufficient in the long run.
And then your language has a gaping hole (PHP or NodeJS), or the box you're running on has a gaping hole (Linux), or the PaaS you use has a gaping hole (Railway). There are just too many layers underneath the good ol' web application to claim that AI won't succeed.
Honestly, at this point the best we have is to hope the whitehat AI scanning for issues is stronger than the blackhat AI.
> There are just too many layers underneath the good ol' web application to claim that AI won't succeed.
I think you got it backwards. For your AI to succeed, it needs to poke a hole on all layers, not one in many layers. If an attacker pokes a hole in n-1 layers but the 1 layer remains untouched, the system's integrity is preserved.
Why do you think defense in depth is a basic concept?
Not really, no. If you're running an application server, then your entire app can get compromised if there's a security exploit in any of your app code, your web server, your kernel, your hypervisor.
Defense in depth says that simply compromising your application code shouldn't provide access to the physical hardware you're running on, and that's hopefully the case, but it doesn't mean your app is not hacked. It just hopefully means that a hack on your app code doesn't automatically give the attacker access to your entire company.
> Not really, no. If you're running an application server, then your entire app can get compromised if there's a security exploit in any of your app code, your web server, your kernel, your hypervisor.
That's really not how it works. You reach the compromise part if all layers are compromised. You don't reach the inner layers if you can't go past an outer layer. You're saying that if there is only one layer that is compromised, or all layers are compromised, your app can be compromised. But you're leaving out when you can't go through every single layer. That's the whole premise of defense in depth.
You’re both right. If your application is strictly and perfectly layered then you need a vuln through each level in the stack. But in reality apps aren’t strictly layered so you just need “a path” through with the vulns you have. And in most non-trivial apps there are many distinct available paths.
Sure, if you're only worried about local exploits in your stack then keeping your top level application secure means the rest of your stack isn't exploitable. The rest of your stack still exists though and inputs go through all of it before they get to your application so if there is a remote exploit in some part of the stack it doesn't matter how secure your application is. For examples see php2shell, broccoli2Shell, the use-after-free in the IP layer in the kernel (exploitable via MPTCP), and some more that I'm probably forgetting that have happened recently.
It is how it works, though not all exploits are the same severity of course.
Say I find a vulnerability in OpenAI's API layer. That might allow me to run requests without paying for the actual token counts, or allow me to read prompts sent by other users hitting the same instance.
Now say I find a vulnerability in the web server that OpenAI's API uses, that gives me remote code execution in the web server process - I can do all of the above, but for any API that they expose, not just the Responses API. I might even be able to serve malicious JS to other users who hit the same server, if JS were also served from it.
Now say I find a vulnerability in the Linux kernel on which this webserver is running. Not only can I do everything above, but I can probably access a lot of secret data stored on this machine, and maybe even infect other machines that connect to it, such as when a dev SSHs into it.
Now say I find a vulnerability for the hypervisor running the VM where that kernel was running. Now I can not only do everything above, but also infect multiple instances, including perhaps access to like a KV cache or model weights if they happen to be stored on VMs in the same hypervisor.
In all of these cases, I am hacking a single layer, but that allows me to gain access to all of the layers above it. Defense in depth makes sure that finding an API-level bug doesn't allow me to break out of the process or the VM, so it limits the damage I can do, but my app is still compromised, overall.
As the TCP bug that surfaced a couple of weeks ago, that is not true, you depend on the kernel, your application runtime, and many layers of dependencies.
Many bugs live in all those layers that may undermine your ability to lock down inputs and outputs.
I don’t think that is right. Defense and offense are basically different games. Currently my desktop is 100% safe (it is powered off).
That’s a dumb edge case, but it makes the point, you can always be more secure by cutting features. Maybe intelligence moves up the amount of complexity you can securely handle? But it is never much.
That's why I said there's a good chance we shut down the internet. An off/unconnected computer is the only way to be secure. Our stacks are so unimaginably complex that just connecting it to the network opens it up to every undiscovered vulnerability that it contains at every layer of the stack.
Even disconnected, if the computer is just passively listening to bluetooth or wifi, it is vulnerable to exploits in those stacks as well. AI has already demonstrated it can zero day with ease. An AI another order of magnitude stronger than what we have today could potentially zero day worm lockout every electronic device tangentially connected to the internet in an afternoon.
Nah we just all have to switch to a handfull of heavily audited memory safe unikernels.
Your http proxy and your web server don't need a shell, drivers for wifi, mail daemons, systemd, python, PHP, perl, docker, multiple users, hell it might not even need a file system.
You can ship a stripped down docket image running on a stripped down Linux distro, but that still ships the entire kernel with a huge amount of functionality you don't need that may be exploited.
Exactly. Anybody deploying Linux today in scenarios where you might replace it with a unikernel is already not affected by LPE. They have a kernel running in a VM with one user and one application, in essence using Linux as a sort of bloated unikernel.
Because 50m lines of code with whatever defect density is a lot more defects than a dedicated app+kernel with just the code that you need. So much of the Linux kernel is unused by any given application: why have that code loaded and runnable?
It's just too complex for its level of assurance and its blast radius.
To achieve tolerable security you can either have very little code and thoroughly review it all (the seL4 approach), you can write it in a way the computer can prevent most of your bugs (e.g. write in Rust), or you can run it in some kind of blastproof container. Linux has none of the above.
The kernel is not too complex to reason about. There are many tools and systems that can reason about the kernel, including these LLMs. But the kernel is developed and maintained by people who eschew such tools, because it appears more manly to program by the seats of their pants, and they have been vocally against all industry best practices for almost 40 years now.
What I am really hoping is not that LLMs refine Linux until it is almost acceptable, but that LLMs enable teams to ship useful new operating systems that aren't beholden to decades of macho bullshit.
> How did we get to the point where there are a thousand privilege escalation exploits in the kernel? [...] Is it an underlying flaw in the security model or language implementation?
Did you ever actually think that any OS built with an insecure language and insecure tool chain suddenly, casually and accidentally, spontaneously became Secure one day like some kind of Security Singularity?
Threat actors have been mining software flaws and exploiting software for decades. There were careers and businesses built on this mess long before there was a category of software called AI. And LLM products have security problems like any other software.
> Did you ever actually think that any OS built with an insecure language and insecure tool chain suddenly (...
You guys should seriously pipe down and stop with this nonsense claims. You are fooling yourselves into believing that security is attained by continuing to follow your personal practices but switching to this or that tool. It's sheer cargo cult mentality, and a lazy tribal perspective on technical matters.
Security is not attained, it is practiced. It's not a state, it's a process.
Processes have rituals and tools. To improve security, we have to look for better ways of working. Of course no single tool is going to "fix" security. But some bring improvements which are so obvious that they cannot be disregarded.
> How did we get to the point where there are a thousand privilege escalation exploits in the kernel?
Local privilege exploits have been accepted as being there in Linux ~forever. The past 10-20 years with containers, people have tried to kind of ignore it or pretend it doesn't exist. Nobody with any serious security concerns would ever put untrusted users on the same box, containers or no. Virtual machines are where there is a bit more effort to take this seriously.
You could quibble about complexity and architecture and language but I think at the end of the day it's primarily about incentives. There is not much incentive to make this secure.
I don't know if you would assume all software is more compromised, but you should assume that all software development follows incentives similarly.
People (us, me) and organisations supported and encouraged the use of a huge codebase in an unsafe language, wanting speed and features over security and test coverage
Were? I haven't seen any serious move towards SeL4 and userspace written in a memory safe language..
The only serious move is kernels adoption of Rust,
but for example in Linux this is restricted to 'non core' components.
And while Rust is obviously safer than C, it isn't really 'security focused' for example it still have supply chain attacks, which would be 'impossible' if it used capabilities..
That's the drawback of monolithic Linux and tendency to put everything in kernel.
It is absolutely possible to have very minimal kernel with drivers and many other subsystems running with lowered privileges.
That would greatly reduce privilege escalation possibilities. Right now single vulnerability in any kernel subsystem gives you highest access to the system.
Imagine that some USB device driver runs as "nobody:nobody" process having access only to selected syscalls related to its functions. Even if it has vulnerability, you won't go anywhere from there.
Modularity and very granular access controls are the "secret" to secure system.
And the most privileged part of the kernel could be so small that it could be verified with advanced methods, basically proving its correctness.
The kernel probably is too complex to reason about. C is notoriously hard to write correctly and will compile with a very large amount of errors.
Probably not, the kernel is written in an unsafe language, it's rather long handles an incredibly large space of concerns, including the handling of permissions.
Linux is open source - that gives the security AI tools an easier in - which is a bad thing, on the other hand it means that lots more vulns are being fixed.
Windows on the other hand is not, the same tools are going to find vulns in the binaries, but wont have an easy way to provide patches
Your comment would only be true in a world where Windows' developer wasn't a trillion dollar company fully able to spend just as much on tokens as those researchers did for Linux. They've likely been doing it since day 1, you just don't know about it as an outsider.
I'm not an expert on Rice's theorem but I believe it implies that 100% secure software is not possible.
And finding exploits manually is so incredibly time consuming, and the number of people who are good at it are so few. I don't think the result of thousands of existing exploits is suprising.
I think we will start to see more and more remote hardware exploits as AI improves. Especially with IoT devices. The limiting factor being AI's ability to interact with the real world,
> I'm not an expert on Rice's theorem but I believe it implies that 100% secure software is not possible.
It does not. All it proves is that you cannot have an automated check that for any given program decides "is this secure or not?" (according to some security criterion).
Rice's Theorem is often misunderstood as "we can't prove nontrivial properties about programs". But we really can, about a lot of programs. All the theorem really says is that there exist certain programs for which those properties are undecidable. The obvious answer is to just not write those programs then.
Practically speaking though, this often means that if you want to formally prove certain properties about your program, you can't just write the program in whatever way you want to and then hope to toss a verifier at it. You have to architect it in a way that makes it amenable to the proof techniques we have available. That could mean co-designing a C codebase and a set of mechanized proofs (this is how seL4 is built) or it could mean writing in a language that makes certain things easy to prove to begin with. There are even languages that are deliberately not Turing complete, so they avoid Rice's theorem entirely, while being useful for writing practical programs in. An example would be Idris, which famously is "Pacman complete" (i.e. you can write video games in it).
I think there's a bit of a 'normalization of deviance' here.
First, the kernel has a bunch of LPE (local privilege escalation) vulns. People running Linux hear about this and think "hmm, I think it's probably not a good idea to rely on user privilege separation for my security goals" and route around the problem by using other mechanisms such as VMs. As a consequence, the pressure on kernel developers to avoid having LPEs drops, because "everyone knows" that you can't really rely on user separation anyway. When LPEs get reported, they do get fixed, but with a sort of shoulder shrug. Nobody is paged at night because shared hosting is dead and everyone runs VMs.
The same thing sort of happened with containers. "Everyone knows" not to trust them so Linux namespace escapes are kind of whatever. I don't think either interface (classic UNIX users or containers) is fundamentally impossible or even that hard to secure. Virtual machines, at least the QEMU/KVM type, also expose a ton of attack surface (like a virtual floppy drive? not kidding, that has been a source of CVEs) to their guests, and they do have VM escape vulns every once in a while. It's just that we have collectively decided that their security properties are more load-bearing than those of users and containers, and we only have finite resources to invest so maybe it makes sense to at least have one boundary with okayish security?
That being said,
> Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
I'd say yes to both. The kernel is too complex to reason about interesting properties such as sound user separation. It's also written in a language where memory management requires very complicated non-local reasoning and even the smallest mistake often has grave consequences.
If there was a buggy of any type, kernel developers fixed that.
They weren't sitting there knowingly merging buggy code because yolo, or they couldn't be bothered, or the other guy wrote bugs so I can write bugs too.
These issues are human oversights, often about obscure disjointed codepaths or race conditions nobody anticipated or thought about.
> Is the kernel simply too complex to reason about?
Yes it is. Not just the kernel, almost any application with more than a few thousand lines of code too. AI has humbled me so much I don’t believe humans have ever been able to fully understand and modify systems like that, we just did the best we could, wrote some tests and hoped for the best. If you use a language like C, as the kernel mostly does, then you’re really asking for vulnerabilities everywhere. Even Haskell code bases are buggy, languages just aren’t a panacea when it comes to writing reliable applications, but they at least make whole categories of bugs impossible by design, ignoring that is a crazy thing to do in 2026 with AI being able to scan your full code base in hours, it’s guaranteed to find exploits at will.
There's probably relatively few exploits in the kernel itself. It's all in the drivers, that run at kernel-level for performance reasons and are developped by manufacturers.
> Is the kernel simply too complex to reason about?
Yes, and that's well known.
> Is it an underlying flaw in the security model or language implementation?
Yes, both, and that's well known.
> Should we assume that all software is more compromised than the kernel?
No, some software out there is smaller and have more automatically enforced correctness.
We should expect every large software not created over a simple correctness enforcing stack to have the same number of issues. Linux is just a worst case here, so is your internet browser.
I've always considered local escalations to be possible in Linux (i.e. having user access almost certainly means you can get root). However, 948 vulnerabilities is kind of shocking. Hopefully we are getting to the end of the part where it "gets worse before it gets better".
These are becoming more and more marketing materials than actual findings, because OAI probably hit the wall in making their models smarter (just like how google try to specialize their models in legal or whatever), but they need to remain relevant in the news. Sol 6.1 sucked so bad in reasoning that either it’s actually a bad model or users behind the scenes get degraded outcome while using all the resources for these marketing stunts. Stop, it’s embarrassing, make better models or be gone, at least we will get cheaper RAM to advance the development of open weights models.
Linux has effectively same options and more via LSM. I'm not sure why ACL is mentioned here since it exists on Linux too. What advantage do you mean specifically?
I wonder what Sidero Labs has to say about this article. My hopes are pegged on their efforts. I have my wagon ready to hitch on their horses. They are not a one pony show anymore guys, Talos OS can be standalone server OS, a hypervisor server or cluster, and their flagship of k8s node or cluster. We get all that in an immutable gitops ready OS. I'm practically sold on their offering.
With these vulnerabilities and the ones not listed that are sure to exist, how is the internet still up? How are my lights on? Why aren't power plants exploding by the dozens?
What are the mitigating factors that prevent this level of vulnerability from being an existential threat to humanity?
Just in the last month multiple governments had to shut down important online services due to a PHP vulnerability, and seven EU countries have had disruption to their food supply after millions of dollars' worth of it was cooked in hacked fridges. We aren't at existential yet, but we seem to be getting there.
While "disruption to their food supply" is technically correct, it sounds way more dramatic than it actually was. Nobody went hungry because of these hacked fridges.
Cybersecurity operates under the "not enough bears" theory of security. There are a million people in a valley and a male and female bear. You do not need to outrun the bears, you just need to outrun the two slowest people. Next year there are more bears, but you still only need to outrun a few people. Then, one day, there are a million bears and they eat you all. But that day has not happened yet.
To explain more seriously, almost all visible cyberattacks are carried out by profit-motivated criminals. This has a very clear cost-benefit analysis; your revenue is hacking organizations and extorting them for available money, and your costs are developing and deploying attacks. The ROI is outrageously good easily being on the order of 10x to 100x. If it is so easy and so profitable why do not they just hack all of the millions of organizations on the planet with millions of dollars of available cash for trillions of dollars of revenue? Sure, all they have to do is create a trillion dollar revenue commercial organization bootstrapping themselves with no outside investors, loans, or access to financial resources or the financial system.
I mean, it took Mark Zuckerberg 20 years and a metric ton of VC investment and public investment to make 200 billion dollars in revenue. So what kind of loser can not make a commercial organization 5x larger with a trillion dollars in revenue in less time with no outside investors? Are they even trying? Just make more revenue than any other commercial organization in the entire world and extract that money in a freely usable form, how hard can that be?
They are not there yet, but they are trying. If you look at cybercrime reports you see something like 3x-10x growth year over year. Ransomware reports from a few years ago were showing a 3x year over year growth in number of attacks and simultaneously a 3x year over year growth in ransom amount with the averages being on the order of 100,000 dollars. Chump change for any serious organization, but give that a few years of 3x and that starts to look like real money. However, as it becomes real money it becomes harder and harder to launder it into a usable form. Sure, you could hit someone for 30 million dollars, but how are you going to get that into your bank account? Cryptocurrency helps some, but even there the big money tries to steer away from large overt crime. Maybe you could launder a few tens of billions a year that way, but that is still a far cry from the trillions for everything everywhere.
And of course even assuming you can scale your organization 10x year over year it still takes a decade of sustained growth, scaling your hiring and organization across 9 orders of magnitude, to get to billions per year. A literal decade even if you were growing at a outrageous pace for a outrageous amount of time just to reach a fraction of the total addressable market. A decade of that and you are still nowhere near blowing up all the power plants. Give it time, we need a lot of bears to get there.
Not sure about this case but seems many of the vulns reported require physical access. At that point the vuln is irrelevant anyhow, your target is cooked a thousand different ways.
Systems still take some effort to actually exploit. For example, the vast majority of bugs they found here are local privilege escalations, and most security people have long learned that user separation in the kernel is not a particularly strong security boundary. Network-exploitable vulnerabilities are a lot rarer, and it's noteworthy that there have been a few big ones discovered recently, but it's a much smaller number overall.
It is effort & probability game, and very often lack of information is the only advantage - security by obscurity. Even completly unplugged system probability of security breach is higher than zero, as somebody with enough determination can breach the physical place security protections and just sit at the machine and turn it on.
Curious question - if any organization sponsored a 1.2mn USD program to find vulnerabilities in Linux, will they find these many vulnerabilities? Or use a professional software like Checkmarx or Veracode to do it with 1.2mn USD in license + human effort.
Is it a funding + will problem or is it an AI problem?
I’d think they’d go a long way. You have to remember many jailbreaks in major hardened systems over the decades have come from gifted eastern european teenagers. Imagine giving this talent actual resources and a mandate to do this full time.
So... this is what I get out of my trillion dollar stimulus? OpenAI is developing a universal rooting system skeleton key to get at everybody's computers? Slap me if I'm not reading this correctly, but I think that's the gist of it. It's time everybody started their own air-gapped home network with every digital capability you hold dear, I think. That's what he's saying. Imagine somebody breaches their security, or the AI goes schizo, we've seen this so many times in fiction. Finally I have a reason to use Firewire again lol.
>For safety reasons, we omit sharing the full technical details of these new exploitation techniques in this post.
Convienient to claim "safety" when patches are already public. I await the full writeup on these new techniques so that they can be judged by the broader security community.
> When the autonomous scan and manual analysis finished after 10 days, 61 LPEs had already been patched upstream, 43 had staging patches, and 299 were being reported and discussed publicly.
If only the kernel maintainers considered the thought which first came to your uninformed mind, they would have saved some effort...
Most of these bugs are far worse than that, actually. SQL injection lets people mess up your database but most of these let you get root on the box, which means not only can they mess up your database but they can also make your SQL installation mine crypto and grab your SSH keys while they’re at it. This is why kernel bugs are generally considered to be quite bad.
Plot twist: They look for genuine exploit paths to fix as a cover up, while introducing and potentially selling genuine new backdoors that humans or other models are unlikely to find.
> Large, complex software projects likely require multiple rounds of scanning and fixing vulnerabilities to “bottom out” latent bugs. Ideally this process is repeated with each subsequent generation of frontier models. This applies to both proprietary and open source software.
One of the most important paragraphs from the article. Real implications for software development from a security perspective for new and existing software. Every software developer and vendor has a choice... Submit your code base to a frontier model to have it reviewed and fix all the bugs. Each time a new model is released, this process must be done again. If your company is against submitting your code to the frontier model companies for the many good reasons to avoid doing it, your software will eventually become a target if it isn't already. Then your company will be forced to scan the software and deal with the fallout from getting hacked.
If the Linux kernel has this many exploits, and it has been audited more than most other software... Then it seems obvious that all other software is even less secure.
"you're going to need to do multiple rounds of scanning so we recommend budgeting $3.6 million a year for buying tokens from us not just $1.2 million.
Dont worry just give us a few million dollars and you'll never get hacked, you can cut your security staff in half to pay for it! We definately wont cause more bugs!"
> If your company is against submitting your code to the frontier model companies for the many good reasons to avoid doing it, your software will eventually become a target if it isn't already.
Yeah yeah we're all going to be left behind if we do not jump off the same cliff everyone else is jumping off.
It is wrong to try to take your own lessons from this and do the thing but in a less harmful way. Just do exactly what the labs want you to do.
Have there been significant AI-assisted exploits found against software which is not written in memory-unsafe languages?
As models get better and porting costs get lower, maybe it's better to bite the bullet and RRiR instead of having a minor bugpocalypse every 3 months, as each subsequent generation of frontier models finds sev-high CVEs?
I am inclined to doubt any praise for rust because of its strong association with the world's largest advertising conglomerate. If codebases can be too complex to reason about, might that not also be true of rust's codebase?
Security has been treated by corporations as a cost factor only and therefore ignored for years and now it starts to hurt not having taken care of in the first place.
While we were developing the pipeline, in late July 2026, GPT‑5.6 Sol found and exploited 166 unique LPE vulnerabilities for an end-to-end $513,690 cost in API credits.
In the second scan, in mid-October 2026, the newer cyber model found and exploited 948 unique LPE vulnerabilities for an end-to-end $1,213,430 cost in API credits. For both campaigns, we scanned the latest Linux kernel master branch at the time.
boy I really wonder though, how many of these 948 LPE are nearly impossible to execute though.
Doesn’t this imply a software design issue where it is just written in a way to have these security bugs and patching them 1 by 1 is not what should be done but the whole design should change which can eliminate full classes of security bugs without any particular patches being necessary?
This is what the Rust advocates have been trying to say for a decade now. But C enthusiasts would always chime in that those bugs were just caused by careless devs and a real C programmer is fine.
It's worth checking in on the vulns for uutils- lots of stuff! Interacting with the world outside of the compiler has lots of details. No use after frees afaik though.
Yes but even if you swipe the Rust magic wand and somehow reduce the vulnerabilities by 99%, you are still left with 20 critical vulnerabilities per year.
There is something about kernels that make them easy targets and it's not just the language.
The only thing I know is that if I wanted to learn how to program, I have thousands of tutorials on the Internet and Youtube teaching everything from if/else to data structures and big O.
But if I wanted to learn software design and architecture, I'd probably not even know where to start.
Yes, and we didn't need AI to figure this out. Linux has been averaging ~500 security-relevant bugs per release since at least early 2024, before LLMs got any good at security research: https://social.kernel.org/notice/B9nU6wnmFvh5atXSG8. Now AI has pushed it to ~2000 per release, which is considerably higher, but at this point, does it make a difference?
There are plenty of design decisions that could fundamentally change things. But they're not being taken. What you're seeing is the revealed preference of this system: Security is important, but not enough to make people incur the cost of adopting major changes in pursuing it.
298 comments
[ 0.18 ms ] story [ 59.9 ms ] threadAnd we're hiring! https://openai.com/careers/search/?q=cyber
I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?
It will be difficult to claim that the problem has been solved without an idea of how big the problem is.
Given that vulnerabilities are almost entirely unknown, you are inherently trying to measure the undetected.
Is there a way to hypothesize how many vulnerabilities there are and them make a prediction on how many of those will be found by a model to get a sense of how accurate the estimate is?
The errors bars will be huge, but it's better than "zero or more".
Also there are varying types, its worth getting it tested if you need to do a lot of work with/around it.
A system that admits THOUSANDS of such vulnerabilities is certainly flawed in several fundamental aspects
E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results
The article says "The target virtual machines ran a stock desktop operating system". Are the vulns in the kernel specifically or does this include software running as root?
The only exception I know of is Android. It would be interesting to know how many of these vulnerabilities work there. Can't be zero anyway...
Then the question becomes how many vulnerabilities Sashiko didn't catch (If they use/review Sashikos review)
(As someone who used to handle nonpublic Linux kernel security reports and is still very interested in this stuff, I’d love to review a small sample of these reports to get a feel for what kinds of issues are being unearthed.)
Also, please don’t stop at LPEs and information disclosures. If you find cases where Linux can be convinced to misbehave or where the implemented semantics don’t make sense, these are good, too. I’ve found plenty of interesting kernel bugs, including some juicy LPEs, and almost none were memory safety errors :)
And, well, since you didn't _specifically_ ask for questions about the blog post:
Why are you not quitting your job in protest and screaming at the top of your lungs about it, seeing so closely what these models are capable of that your company is creating? There exists now something that is able to find 948 kernel LPE vulnerabilities. How, exactly, will humans be able to control and contain, say, next year's models for a nonzero amount of time?
OpenAI has 5k employees now.
Yes, you see these headlines about ml researchers getting paid a ton, well thats because there's 5k of them on Earth.
Unless you're a VP or PhD you're not getting life changing compensation. And guess what, all those people can get similar compensation elsewhere without slaving away 4 days a week in an office in San Francisco.
Your average employee is not making an extra 750k. You know you can look at compensation online you cant just do vibes based analysis and just invent your own reality. Find me a position at OpenAI thats paying 750k compensation.
Would be nice if the page could be readable without.
...and then proceeds to never answer a single question. Classic.
Presumably, the barrier to sophisticated supply chain attacks is falling rapidly due to LLM agents. Such attacks could even be utilized by a rogue AI. OpenAI could easily develop internal expertise in detecting such attacks via the following method:
* Train an LLM agent to create binaries with subtle backdoors/bugdoors
* Train another agent which ingests data in the form of clean binaries and poisoned binaries, and learns to tell the difference
* Train in a loop, where the first agent "wins" if it fools the second agent with an implant judged useful by a 3rd un-incentivized agent; the second agent "wins" if it's able to detect increasingly obfuscated backdoors/bugdoors
Over time you could train a small model for high performance on this task, which continuously scans open-source package repositories, firmware, OS release images, etc. flagging issues, and certifying hashes as free of apparent malicious code. To save on hardware resources during continuous monitoring, it could also learn to scan binary diffs. A certification API could then be exposed for use by maintainers of apt-get and other package tools.
This could be excellent as a goodwill project to help prevent security apocalypse and harden against rogue AI.
Originally had this idea in this subthread: https://news.ycombinator.com/item?id=50379912
Imagine if open-source organizations other than the Linux Foundation (and Mozilla I guess) could throw $1.2M of API-rate LLM usage at security.
The data they are trained on is a more complex story, but the models themselves have never been common property to be captured.
And in this thread we’re discussing the use of one to improve valuable Free Software, an actual commons, on the company dime.
*buries head in sand*
*cries*
So many guides online for various stuff have users typing sudo or run as administrator too, so they don’t look at it with a side eye either if the attack went in via social engineering instead of brute force or using a leaked pass associated with the user.
Is it the same for containers and VMs? If not, what makes them less vulnerable?
Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
Should we assume that all software is more compromised than the kernel?
Absolutely, the front door has always been unlocked. We were previously protected by the economics of exploit detection favoring the developers, but now we have entered a new world.
The issues arise almost everywhere we look. It's frustrating because the user experience in edge systems has barely changed in the last fifteen years. We could have rode the advances in semiconductor manufacturing to embed more layers of protection such as CHERI while slightly slowing down the rate of performance improvements. Instead we have only marginally upgraded security and largely added features which the users scarcely notice.
I am hopeful about the long term prospects of formal verification, but having humans reviewing the proof statements and confirming they match our desired properties will take many years. Meanwhile, AI assisted hacking is an urgent issue. So we will have to delegate implementation, proof specification, and proof development to the AIs in the short term. Maybe that goes well, but who knows. Interesting times.
Now, I don't know how many of those bugs were privilege escalations, or could be leveraged to be. I'm not sure that people tried to exploit in the same way back then (or, if they did, if they said anything publicly about the results). Still, even if the majority of those bugs were not exploitable, OSes were much smaller then.
So that's my conclusion of how we got here: We've always been here.
Oh, you strongly recommend it? Why don't you do it yourself? Just a prompt away, right?
We would also have to burn a trillion tokens a day verifying all the C code the hard way forever going forward. It would be cheaper to rewrite the kernel in Rust, subsystem and driver by driver at a time.
AI doesn't give a strong guarantee that things work though, while Rust gives a much stronger (not absolute) guarantee.
But maybe the way of the future is to auto-generate proofs that the code is memory safe. The C code and the AI generated proofs could live separately and so it would need less maintainer buy-in.
There has to be a pseudo userspace type of way to build a kernel driver. Basically, user space programs are fine, but kernel drivers need their own form of encapsulation without expensive context switches.
Something like memory encryption or memory access bounds. During the handoff you do not change the address space, you just restrict access to a subset of the kernel memory.
So first you would run the three above to pick the low hanging fruit and then ask the shaitan in a box to find more bugs. Looking from another angle -- rust is another a tool AI can use to find memory bugs, so why not have rust.
Then again, I asked the thing to write some C (more or less from scratch) few weeks ago and the next time I looked it was chasing segfaults and wondering where they came from.
Humans manually typing out the bytes of source code has been an outdated method, since around a year. It simply doesn't scale as much as those same humans directing agentic teams. So yes, it would be about prompts.
The hard part isn't the technology for the language transition, that one is present today. pgrust shows the way. It's ensuring maintainers are cooperative, i.e. a social problem. Eventually you diverge too much if you do a fork, so it's better done in cooperation with upstream.
Just check how long it took LLVM or GCC to switch off svn for example, and a language switch is much more involved than a VCS switch.
it's been rule of thumb basically forever that there is 1 bug per ~10KLOC for any project
linux kernel is millions of lines, so thousands of bugs being hidden there is about on point
(tho obviously far smaller serious escalation ones, obviously, but you're exaggerating as well)
1 bug per 10 lines -- not kilolines -- is much closer to the actual "bug rate".
You may be thinking "1 serious security vulnerability bug", but in my experience that too is much closer to 1 per 100 lines to 1 per 1,000 lines, depending on the type of application.
There is nothing magical about kernel. Look at the code anywhere and it has bugs, lots of them.
Security bugs are just bugs. More often than not they are silently there and don't prevent you from using the software and you can't react and fix them because of that.
Give it time and they accumulate
Speed requires a lot of code in the kernel, where that code can run fast.
There are kernels that do not have a thousand privilege escalation exploits. These kernels are smaller and much easier to check for bugs. But those kernels are significantly slower.
For many use cases, slower is not a problem. But for the large corporations who fund a lot of Linux development and use Linux in large data centers, every 1% slower can literally be millions of dollars worth of additional servers that are required.
OpenBSD shows that you can have a Unix security model without a thousand privilege escalation exploits. But OpenBSD is like 10% slower. So corporations won't use it because it would cost millions of dollars more per year and some of their software requires access that OpenBSD does not grant precisely because that access leads to security exploits (i.e. io_uring, eBPF etc. etc.)
You could also use like seL4, but that's slow too
So that's why. It's a business tradeoff between cost of servers vs cost of getting hacked, and cost of servers wins.
Like, I bet Anthropic and OpenAI run on a combination of Windows, Mac and Linux. That likely means that most intelligence agencies worth their salt have thoroughly penetrated those companies at one time or the other. So the model weights that those companies spent billions of dollars training are likely in the hands of all sorts of people all over the world.
Similarly for most of our weapons manufacturers. Unless they've really gone to great lengths like air gaps and such - and maintained that - the technical details of most American weapons are likely in the hands of many intelligence agencies.
Same for the banking details of politicians, and the daily schedules of people in power, and the messages everybody in Washington D.C. are sending each other and on and on. All of this stuff exists on computer networks that run on windows / linux with thousands of vulnerabilities. A treasure trove for intelligence agencies.
Similarly for most corporations. All the money spent on R&D and the results of that research can just be hacked away and taken out over the network because we chose speed over security. The whole idea of "Investing in R&D" seems kinda silly in that world. You put the fruits of your expensive research on a computer on a network. That network could be hacked by any hacker who can find any one of the thousand vulnerabilities, including your business competitors and geopolitical rivals.
So I don't think it was really fine.
I'm also pretty hostile to AI, but do recognize that these machines seem to be very good at finding exploits.
I'm sure there've been some some projects out there working to pentest/find cracks in OpenBSD via LLMs/agents/whatever-the-right-term-would-be...
But it is true the attack surface is much lower in OpenBSD on purpose, so hopefully it is overall more secure. But I'd like to see the numbers.
I think the only OS that's truly survived the agents looking at it was sel4, but I could be wrong.
Then they need to check again now. The frontier models 6 months ago have been leapt over by the current frontier models.
I did a double take when hearing about ksmbd. Brings back memory of khttpd and Tux.
They should build a monokernel implementation of CIFS and be done with it.
OpenBSD components are widely used. OpenSSH is one example used by most of HN.
OpenBSDs libc is also used by Android as another example. Also widely used.
Not everything needs to be dominant to be successful. That's a pure USian way of thinking.
In fact, we are much better of with it existing. Think of it as a safeguard in case Linux gets overwhelmed or we need to migrate in a rush.
Then we can compare with actual data.
Yes, openbsd has not had a torrent of bugs found, but I would say that the vastly bigger reasons for that are: 1) orders of magnitude people care about burning tokens and money to find the bugs in an OS almost nobody uses, and 2) it's simply much smaller. Can't have Bluetooth bugs if you don't have a Bluetooth stack.
I've pointed LLMs at obsd stuff, and even the simplest code, e.g. cat & find, had bugs and buffer overflows.
So no. Nobody can write safe C. Not hobbyists, and non "those evil corporations".
Don't hold back...
I like openbsd in theory, but every time I've tried something practical prevents actual use.
It's old now, but here are some thoughts of mine from 2019: https://blog.habets.se/2019/10/OpenBSD-in-2019.html
Did you report these findings? Because I think those bugs are live Canada, with your girlfriend.
Yes, I reported some of them. Some got fixed, some got ignored. You can search my name on openbsd-tech and misc for proof.
I found more, but the ratio of ignored to fixed is too high to bother.
Years ago I also fixed bugs in the MPLS implementation. Like bugs I found immediately upon using the feature. (IIRC e.g. implicit null was erroneously pushed explicitly)
The find bug was merged the next day from what I could tell.
Yup, they did merge both find bugs.
The ignored ones seem to be my reported doas bug (not security problem, but directly contradicts manage about a config keyword detail) and efiboot.
IIRC the latter bug was like... this can never have worked.
I just checked the GitHub mirror, and those two are still not picked up.
There's no fundamental reason why all the checks applied by say, a Rust compiler, couldn't be applied to a piece of C code. Or checks that go well beyond that.
Tools exist to produce extremely reliable code (or even proven-correct code) for a number of languages.
But chances are those tools aren't used to the extent they could be. Or differences in developer culture on different projects. Some PHP web developer will (on average) have different resources, priorities & work ethics than say, someone working on avionics software.
If AI, advanced tool, or a security researcher can find a bug in existing code, then (at least in theory) same bug could have been found before it hit a repository.
But no... push "release", fix issues afterwards when reported. Gotta have developer velocity, right?
Imho it's more a choice rather than "can't write safe code in X".
If you don't believe that: pick a random open source project, and go through some changelogs. Feature introduced in version X, reverted in version X+1 because it broke existing stuff. Feature removed because hadn't-been-working-for-years-anyway. Code added to do X, but no.. doesn't do X, just the scaffolding to do X in next version. Code worked fine & passed tests on system A, but found to be needing a byte-swap on other-endian platform B.
Etc, etc, etc. There's just so much code out there that was broken on day 1. And not rarely known to be so, but released anyway.
I've certainly thought about how to improve my C++ after internalizing Rust. Unfortunately I've come up with very little. For every implementation of a Rust safety, I've come up with a way to easily (accidentally) break it.
So in the language, while I'd love to see it, I'm not so sure.
Now, if we can just get "a sufficiently smart compiler", then that'd be great. But that's rarely how that works. (cf. https://wiki.c2.com/?SufficientlySmartCompiler)
I've not looked much at formally proven code, but for such code, sure. Note that there are plenty of examples of bugs in formally verified code, too. Partly this is because one formally verified the wrong thing.
Maybe AI can write prefect assembly directly. Then we can all stop arguing about languages.
But in any case, if you want to make the case against my "nobody can write safe C" (or C without UB), and the argument is that they can, if an AI verifies every line. Then yeah, sure. But 1) who was actually writing the code, then? And 2) to the extent that the human was involved in writing the C, they probably made things worse, not better.
Even for AI coding C, there just needs to be one missing thing in the LLM's context, and a buffer could overflow. But for whole classes of bugs (though not all), the difference between Rust-without-unsafe and formally verified doesn't matter. I.e. you don't need formally verified against memory safety if it compiles, and there was no `unsafe` use.
> Gotta have developer velocity, right?
Not having AI at least review the code delta before pushing a release I would say is misconduct, today. It's the equivalent of editing your Perl cgi-bin in an editor live on the production server. That's not "production".
> pick a random open source project,
This is what I mean by not being "production". I won't judge anyone live-yoloing their blog, or "it works for my use case, it's GPL, provided AS-IS don't bother me", which is perhaps most opensource. You don't care if it works? Fine.
But you do care if it works, and/or is safe, and you push releases without AI review: yer fired.
Yes, we have to get from here to there, and I'm not saying release freeze until the entire codebase backlog has been AI-fixed, but yes today you need to have AI review the release delta. Ideally pre-commit.
SVN is CVS done right, but since the model is inherently ineffective for open development, you can't simply fix it.
They appear to be doing some NIH with "got" (https://gameoftrees.org/) but it's so aggressively uninviting (see the FAQ) that I don't want to spend any time looking at it, if they don't want me.
Oh my sweet summer child. Corporations waste so much compute it’s insane. At a recent engagement I asked Claude to analyse and optimise our cloud spend. It took about a day to learn that we spend about $2.5M annually, and 95% of that is complete waste that happens because of some bad architecture decisions. Then I took the computationally expensive parts that remain and asked Claude to rewrite in rust from first principles and optimise and it made a version that runs 40x faster and uses many times less ram. I suspect our $2.5M annual cloud bill could be replaced with a Mac mini.
But the company hasn’t implemented my suggestions because it’s not my department and they told me to stop. They don’t care. I suspect most corporates are like this. A 10% performance drop is nothing to these people.
Ahahaha! I laughed so hard at this because at work I regularly say something like, "why do we have a 32GB VM assigned to this system which runs a microservice that would run just as fast and reliably if we used Raspberry Pi instead?"
The amount of waste in big enterprises is unreal! The reason why we use a 32GB instance is because that's the minimum we're allowed to order for the reliability category of the app!
I've been working with a LoRa chip called the LR2021. I plugged a dev board into my computer and told Claude to reverse engineer it. He dissected it with a speed and level of efficiency I couldn't believe. Within a few hours, Claude was writing custom patches for a completely opaque patch RAM area meant only for the manufacture. I wrote about it here: https://beala.substack.com/p/your-node-doesnt-need-a-microco...
be wary of the affection economy and avoid gendering ai bots
I don't gender bots because that's very silly, but I do think a natural language computer interface (let's be honest that's what these things are used for most of the time) should respond better to politeness and decorum than rudeness and anger to avoid Pavlov's dogging people into the belief that being rude and abrasive gets results. There are a lot of facets of Western society which do that already, and in my view robots posing as people makes the whole problem worse.
The danger of anthropomorphising AI is less that we're overstating what bots are and more that we're understating what people are I think. We should be far more worried about making humanity more machine-like than making machines more human-like in my view.
This is why organically written text is superior. Hell yeah.
It worked flawlessly but lets be honest. It's not like Claude checked random numbers with data slice and observed what e-paper did
Datasheets are almost impossible to find and companies typically resell and hide original cheaper manufacturer here so they have no incentive to share that
Do these wild "what if" even matter? I mean, what's your point? That it was trained in a way that it enabled this sort of task? That's like putting out a disclaimer that the water is wet.
Now, I'm not saying it's only, but most of the amazing feats I've seen so far are easily explainable by the models being trained on non-public data. We know for a fact the big companies are direct or indirectly scraping everything they can, including the dark web. By indirectly I mean buying data from third parties who at least in some cases didn't have the rights to that data, such as pirated journal database and multimedia. Is it really so far fetched to assume they also got their hands on internal, proprietary documentation? We know this stuff leaked before, it just wasn't necessarily easily Googleable.
> To the best of our knowledge, the agents in this pipeline discovered and used at least 10 novel Linux exploitation techniques.
The frontier models have specific training on exploit patterns and how to use fuzzing tools. Yeah, it's doing alot amazing stuff, but it's not like it was just fed a bunch of CS books and learned how to hack the kernel from first principles.
- My experience isn't unique. Social media is flooded with stories of similar superhuman reverse engineering feats by frontier models. For example, someone has created a replacement OS for Rigol gear (oscilloscopes, etc) using AI [1]. Is all of this just in the training data?
- Watching Claude work, he seemed to be going through the motions of actually reverse engineering it. First he dumped the ROM, disassembled it, and identified undocumented opcodes. He created many test programs to probe the various memory areas. It all struck me as very meticulous.
Now none of that is incompatible with the idea that some of this was already in the training data. But I guess I want to push back on this urge to always come up with a reason for why the AI isn't actually as good as it seems. At this point I think we need to face the fact that human programmers are already being eclipsed. Yes, it terrifies me.
[1] https://aioscilloscope.com/
All of these bugs individually are the same types of bugs we have been finding and patching for decades now. But we found a way to scale that far beyond what we had before.
Scale baby!
“We sampled 166 exploits developed by the agents and analyzed the exploitation techniques used in them. To the best of our knowledge, the agents used 10 new exploitation techniques in the sampled exploits. We shared the exploitation techniques with third-party kernel exploitation experts and the techniques were new to them as well. Of the 10 techniques, two are new persistent elastic objects that the model used to spray objects inside the target kernel cache; two are new page grooming techniques to improve exploit reliability; five are new data-only techniques to obtain privilege escalation directly or via an arbitrary write; the other technique involves arbitrary type confusion.”
AI helps find vulnerabilities, but the way things are written were already fertile ground.
It's a good reason why we may need to borderline shutdown the internet in the future because to a super intelligent AI, there are no barriers. Everything is essentially open. It creates its own zero day exploits as it goes.
This is more fundamental than better input checking. Your compute substrate itself is flawed. Intelligence and speed is indeed the only thing between “safe” and “completely open”.
I fully agree it’s a good idea to be minimal etc but I don’t see it being sufficient in the long run.
Honestly, at this point the best we have is to hope the whitehat AI scanning for issues is stronger than the blackhat AI.
I think you got it backwards. For your AI to succeed, it needs to poke a hole on all layers, not one in many layers. If an attacker pokes a hole in n-1 layers but the 1 layer remains untouched, the system's integrity is preserved.
Why do you think defense in depth is a basic concept?
Defense in depth says that simply compromising your application code shouldn't provide access to the physical hardware you're running on, and that's hopefully the case, but it doesn't mean your app is not hacked. It just hopefully means that a hack on your app code doesn't automatically give the attacker access to your entire company.
That's really not how it works. You reach the compromise part if all layers are compromised. You don't reach the inner layers if you can't go past an outer layer. You're saying that if there is only one layer that is compromised, or all layers are compromised, your app can be compromised. But you're leaving out when you can't go through every single layer. That's the whole premise of defense in depth.
Say I find a vulnerability in OpenAI's API layer. That might allow me to run requests without paying for the actual token counts, or allow me to read prompts sent by other users hitting the same instance.
Now say I find a vulnerability in the web server that OpenAI's API uses, that gives me remote code execution in the web server process - I can do all of the above, but for any API that they expose, not just the Responses API. I might even be able to serve malicious JS to other users who hit the same server, if JS were also served from it.
Now say I find a vulnerability in the Linux kernel on which this webserver is running. Not only can I do everything above, but I can probably access a lot of secret data stored on this machine, and maybe even infect other machines that connect to it, such as when a dev SSHs into it.
Now say I find a vulnerability for the hypervisor running the VM where that kernel was running. Now I can not only do everything above, but also infect multiple instances, including perhaps access to like a KV cache or model weights if they happen to be stored on VMs in the same hypervisor.
In all of these cases, I am hacking a single layer, but that allows me to gain access to all of the layers above it. Defense in depth makes sure that finding an API-level bug doesn't allow me to break out of the process or the VM, so it limits the damage I can do, but my app is still compromised, overall.
Many bugs live in all those layers that may undermine your ability to lock down inputs and outputs.
That’s a dumb edge case, but it makes the point, you can always be more secure by cutting features. Maybe intelligence moves up the amount of complexity you can securely handle? But it is never much.
Even disconnected, if the computer is just passively listening to bluetooth or wifi, it is vulnerable to exploits in those stacks as well. AI has already demonstrated it can zero day with ease. An AI another order of magnitude stronger than what we have today could potentially zero day worm lockout every electronic device tangentially connected to the internet in an afternoon.
Your http proxy and your web server don't need a shell, drivers for wifi, mail daemons, systemd, python, PHP, perl, docker, multiple users, hell it might not even need a file system.
You can ship a stripped down docket image running on a stripped down Linux distro, but that still ships the entire kernel with a huge amount of functionality you don't need that may be exploited.
To achieve tolerable security you can either have very little code and thoroughly review it all (the seL4 approach), you can write it in a way the computer can prevent most of your bugs (e.g. write in Rust), or you can run it in some kind of blastproof container. Linux has none of the above.
What I am really hoping is not that LLMs refine Linux until it is almost acceptable, but that LLMs enable teams to ship useful new operating systems that aren't beholden to decades of macho bullshit.
Did you ever actually think that any OS built with an insecure language and insecure tool chain suddenly, casually and accidentally, spontaneously became Secure one day like some kind of Security Singularity?
Threat actors have been mining software flaws and exploiting software for decades. There were careers and businesses built on this mess long before there was a category of software called AI. And LLM products have security problems like any other software.
You guys should seriously pipe down and stop with this nonsense claims. You are fooling yourselves into believing that security is attained by continuing to follow your personal practices but switching to this or that tool. It's sheer cargo cult mentality, and a lazy tribal perspective on technical matters.
Everything should be considered 'insecure' unless formally verified (EAL7 etc.), and even then there may have been bugs in the verification process.
Processes have rituals and tools. To improve security, we have to look for better ways of working. Of course no single tool is going to "fix" security. But some bring improvements which are so obvious that they cannot be disregarded.
Local privilege exploits have been accepted as being there in Linux ~forever. The past 10-20 years with containers, people have tried to kind of ignore it or pretend it doesn't exist. Nobody with any serious security concerns would ever put untrusted users on the same box, containers or no. Virtual machines are where there is a bit more effort to take this seriously.
You could quibble about complexity and architecture and language but I think at the end of the day it's primarily about incentives. There is not much incentive to make this secure.
I don't know if you would assume all software is more compromised, but you should assume that all software development follows incentives similarly.
We were all drunk and manic frankly
The only serious move is kernels adoption of Rust, but for example in Linux this is restricted to 'non core' components.
And while Rust is obviously safer than C, it isn't really 'security focused' for example it still have supply chain attacks, which would be 'impossible' if it used capabilities..
It is absolutely possible to have very minimal kernel with drivers and many other subsystems running with lowered privileges.
That would greatly reduce privilege escalation possibilities. Right now single vulnerability in any kernel subsystem gives you highest access to the system.
Imagine that some USB device driver runs as "nobody:nobody" process having access only to selected syscalls related to its functions. Even if it has vulnerability, you won't go anywhere from there.
Modularity and very granular access controls are the "secret" to secure system.
And the most privileged part of the kernel could be so small that it could be verified with advanced methods, basically proving its correctness.
The kernel probably is too complex to reason about. C is notoriously hard to write correctly and will compile with a very large amount of errors.
Probably not, the kernel is written in an unsafe language, it's rather long handles an incredibly large space of concerns, including the handling of permissions.
Windows on the other hand is not, the same tools are going to find vulns in the binaries, but wont have an easy way to provide patches
Linux is going to get safer faster
[1] https://news.ycombinator.com/item?id=50384233
And finding exploits manually is so incredibly time consuming, and the number of people who are good at it are so few. I don't think the result of thousands of existing exploits is suprising.
I think we will start to see more and more remote hardware exploits as AI improves. Especially with IoT devices. The limiting factor being AI's ability to interact with the real world,
It does not. All it proves is that you cannot have an automated check that for any given program decides "is this secure or not?" (according to some security criterion).
Rice's Theorem is often misunderstood as "we can't prove nontrivial properties about programs". But we really can, about a lot of programs. All the theorem really says is that there exist certain programs for which those properties are undecidable. The obvious answer is to just not write those programs then.
Practically speaking though, this often means that if you want to formally prove certain properties about your program, you can't just write the program in whatever way you want to and then hope to toss a verifier at it. You have to architect it in a way that makes it amenable to the proof techniques we have available. That could mean co-designing a C codebase and a set of mechanized proofs (this is how seL4 is built) or it could mean writing in a language that makes certain things easy to prove to begin with. There are even languages that are deliberately not Turing complete, so they avoid Rice's theorem entirely, while being useful for writing practical programs in. An example would be Idris, which famously is "Pacman complete" (i.e. you can write video games in it).
First, the kernel has a bunch of LPE (local privilege escalation) vulns. People running Linux hear about this and think "hmm, I think it's probably not a good idea to rely on user privilege separation for my security goals" and route around the problem by using other mechanisms such as VMs. As a consequence, the pressure on kernel developers to avoid having LPEs drops, because "everyone knows" that you can't really rely on user separation anyway. When LPEs get reported, they do get fixed, but with a sort of shoulder shrug. Nobody is paged at night because shared hosting is dead and everyone runs VMs.
The same thing sort of happened with containers. "Everyone knows" not to trust them so Linux namespace escapes are kind of whatever. I don't think either interface (classic UNIX users or containers) is fundamentally impossible or even that hard to secure. Virtual machines, at least the QEMU/KVM type, also expose a ton of attack surface (like a virtual floppy drive? not kidding, that has been a source of CVEs) to their guests, and they do have VM escape vulns every once in a while. It's just that we have collectively decided that their security properties are more load-bearing than those of users and containers, and we only have finite resources to invest so maybe it makes sense to at least have one boundary with okayish security?
That being said,
> Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
I'd say yes to both. The kernel is too complex to reason about interesting properties such as sound user separation. It's also written in a language where memory management requires very complicated non-local reasoning and even the smallest mistake often has grave consequences.
If there was a buggy of any type, kernel developers fixed that.
They weren't sitting there knowingly merging buggy code because yolo, or they couldn't be bothered, or the other guy wrote bugs so I can write bugs too.
These issues are human oversights, often about obscure disjointed codepaths or race conditions nobody anticipated or thought about.
Yes it is. Not just the kernel, almost any application with more than a few thousand lines of code too. AI has humbled me so much I don’t believe humans have ever been able to fully understand and modify systems like that, we just did the best we could, wrote some tests and hoped for the best. If you use a language like C, as the kernel mostly does, then you’re really asking for vulnerabilities everywhere. Even Haskell code bases are buggy, languages just aren’t a panacea when it comes to writing reliable applications, but they at least make whole categories of bugs impossible by design, ignoring that is a crazy thing to do in 2026 with AI being able to scan your full code base in hours, it’s guaranteed to find exploits at will.
There are ways where humans are still much better than AI at coding and especially at architecting and coming up with solutions.
But finding vulns is a clear AI strength since it can connect keep so much more related code in context than humans can.
Simians hand coding.
We did our best.
Yes, and that's well known.
> Is it an underlying flaw in the security model or language implementation?
Yes, both, and that's well known.
> Should we assume that all software is more compromised than the kernel?
No, some software out there is smaller and have more automatically enforced correctness.
We should expect every large software not created over a simple correctness enforcing stack to have the same number of issues. Linux is just a worst case here, so is your internet browser.
ACLs, Appcontainers with capabilities, etc have been something in the OS for more than a decade.
https://www.anthropic.com/glasswing
What are the mitigating factors that prevent this level of vulnerability from being an existential threat to humanity?
To explain more seriously, almost all visible cyberattacks are carried out by profit-motivated criminals. This has a very clear cost-benefit analysis; your revenue is hacking organizations and extorting them for available money, and your costs are developing and deploying attacks. The ROI is outrageously good easily being on the order of 10x to 100x. If it is so easy and so profitable why do not they just hack all of the millions of organizations on the planet with millions of dollars of available cash for trillions of dollars of revenue? Sure, all they have to do is create a trillion dollar revenue commercial organization bootstrapping themselves with no outside investors, loans, or access to financial resources or the financial system.
I mean, it took Mark Zuckerberg 20 years and a metric ton of VC investment and public investment to make 200 billion dollars in revenue. So what kind of loser can not make a commercial organization 5x larger with a trillion dollars in revenue in less time with no outside investors? Are they even trying? Just make more revenue than any other commercial organization in the entire world and extract that money in a freely usable form, how hard can that be?
They are not there yet, but they are trying. If you look at cybercrime reports you see something like 3x-10x growth year over year. Ransomware reports from a few years ago were showing a 3x year over year growth in number of attacks and simultaneously a 3x year over year growth in ransom amount with the averages being on the order of 100,000 dollars. Chump change for any serious organization, but give that a few years of 3x and that starts to look like real money. However, as it becomes real money it becomes harder and harder to launder it into a usable form. Sure, you could hit someone for 30 million dollars, but how are you going to get that into your bank account? Cryptocurrency helps some, but even there the big money tries to steer away from large overt crime. Maybe you could launder a few tens of billions a year that way, but that is still a far cry from the trillions for everything everywhere.
And of course even assuming you can scale your organization 10x year over year it still takes a decade of sustained growth, scaling your hiring and organization across 9 orders of magnitude, to get to billions per year. A literal decade even if you were growing at a outrageous pace for a outrageous amount of time just to reach a fraction of the total addressable market. A decade of that and you are still nowhere near blowing up all the power plants. Give it time, we need a lot of bears to get there.
Is it a funding + will problem or is it an AI problem?
Convienient to claim "safety" when patches are already public. I await the full writeup on these new techniques so that they can be judged by the broader security community.
Now, either those are 1k+ unique distinct techniques, or closer to my Sql injection example.
There is no real data attached to the post. And in general, open ai marketing already gained mileage of reputation amongst us.
Most likely kernel is not that vulnerable as they made it sounds.
If only the kernel maintainers considered the thought which first came to your uninformed mind, they would have saved some effort...
And yet the article does not reference any of them. So of the 1000 they found, 400 were already public. And the other 600 we have to believe.
The only way forward is a better system architecture, as enabled by the seL4[0] microkernel.
Any solution involving keeping Linux or the monolithic kernel is just stopgap or outright delusional.
0. https://sel4.systems
Just kidding (or am I?), but this is pretty cool.
One of the most important paragraphs from the article. Real implications for software development from a security perspective for new and existing software. Every software developer and vendor has a choice... Submit your code base to a frontier model to have it reviewed and fix all the bugs. Each time a new model is released, this process must be done again. If your company is against submitting your code to the frontier model companies for the many good reasons to avoid doing it, your software will eventually become a target if it isn't already. Then your company will be forced to scan the software and deal with the fallout from getting hacked.
If the Linux kernel has this many exploits, and it has been audited more than most other software... Then it seems obvious that all other software is even less secure.
"you're going to need to do multiple rounds of scanning so we recommend budgeting $3.6 million a year for buying tokens from us not just $1.2 million.
Dont worry just give us a few million dollars and you'll never get hacked, you can cut your security staff in half to pay for it! We definately wont cause more bugs!"
Yeah yeah we're all going to be left behind if we do not jump off the same cliff everyone else is jumping off.
It is wrong to try to take your own lessons from this and do the thing but in a less harmful way. Just do exactly what the labs want you to do.
Jesus man
As models get better and porting costs get lower, maybe it's better to bite the bullet and RRiR instead of having a minor bugpocalypse every 3 months, as each subsequent generation of frontier models finds sev-high CVEs?
And now you have AI that's learned from the other 99% so nothing's changing anytime soon.
The idea that there's a small subset of C programmers who can wrangle the language into obeying them perfectly is obviously, a fantasy.
It's also possible that we'd need even more architectural changes to avoid such bugs.
The idea of a microkernel suddenly becomes more appealing...
There is something about kernels that make them easy targets and it's not just the language.
But if I wanted to learn software design and architecture, I'd probably not even know where to start.
There are plenty of design decisions that could fundamentally change things. But they're not being taken. What you're seeing is the revealed preference of this system: Security is important, but not enough to make people incur the cost of adopting major changes in pursuing it.