Ah yes. Everyone eventually dies, we just shorthen the wait.
This is like telling that bank vaults always were breachable, and giving out explosives to everyone just made them get breached sooner.
(edit: fact of existance of a vulnerability is not a binary yes/no. the amount of effort needed to exploit even known vulnerabilities is usually staggering, requiring months from very capable individuals with very specific skillset.)
I genuinely would like to know your opinion on what the next steps would be then? If every single exploit was available tomorrow accompanied by a working exploit to operationalize it, or combine it in a chain to compromise any software that has such a vulnerability, would you post it on the internet?
I know you a bit by reputation so that was my assumption. I assume you mean responsible disclosure? I thought your statement was implying otherwise, maybe the unsaid implication was against the "extortion" argument?
In any case, what I want to say is this: All technology has tradeoffs but this has shown we are locked in a game theory mechanic not dissimilar to extortion AND the security implications are part of that.
- We have to build AI or someone else will.
- You need our tools to protect yourself from our tools or similar ones
- You need to use AI or someone using AI will take your job.
Many examples but it is very "i created the disease to sell you the cure". If this were any other technology, we would expect more reasonable mitigations, extraneous factors like software bugs existing aside. It speaks to a fundamental lack of control on the part of the labs that we take for granted: they simply don't have control of the PRODUCT, much less how it is even used.
I cannot help to think that the idea of go fast and break things had something to do with the current state of affairs. I would say that the next step should be slow and consolidate, but it seems like we are not heading in that direction.
At this point I don't know what will be worse for the global economy the AI Bubble bursting, or the gigantic crash all this vulns might bring if they force the stop because we went too far and everything came crumbling down...
> The vulnerabilities exist whether or not the AI does.
No. Even if we ignore the people newly enabled to slop out vulnerable software without having a clue about security, the AI powered supply chain hacks inserting vulnerabilities, the influx of AI contributions that drown out improvements in FOSS security, and the industry pressure to ship faster while reducing quality or be left behind -- all because of AI -- security through obscurity has historically been the most effective security.
I don't think your argument holds under theory or (especially) practice.
The vulnerabilities we are seeing in the news are, as a rule, not things introduced by AI. That could change, but this rebuttal isn't on point currently.
> security through obscurity has historically been the most effective
Security through mathematical impossibility is the most effective. AI-assisted brute-force vulnerability search is revealing that what we considered obscure a year ago is no longer hidden and was never secure.
Whatever you can do with the latest OpenAI and Anthropic models today you’ll be able to do with Chinese open source models three months later, with far fewer restrictions. Putting your head in the sand and blaming a single person isn’t going to make the problem disappear, as so many company IT leaders are finding out.
We are where we are, I think most people here understand that. However, if where we are is the right now is just the starting line, and it is already shaking the foundation of all modern society, then I think it is right to be concerned; parent comments quip aside.
There are multiple philosophies regarding this question, and especially historically people were in disagreement. Many companies used to be angry when attacked or probed by white hats. They thought "yes I suppose I should lock my door, but if someone is out trying door handles they should be put in jail". I think there is some logic to that perspective. But clearly that philosophy ended up losing to the "victim blaming" one where everyone is responsible for securing their systems.
Since we are entering a new age, maybe it's time to revisit the question. But ultimately I think we will land in the same place, in fact even firmer than before.
So why don't we just lock up all the whitehats and people behaving suspiciuosly and probing things? Why can't we achieve cybersecurity in this way when it seems to be a common thing in physical security?
And the answer I think is quite simply that there are too many attackers and they are too far away. When the russians and chinese come knocking there isn't much we can do to shoo them away. Their governments will not cooperate in detaining the criminals. Entirely blocking such countries from the internet isn't viable either. And that's why we arrived where we did. Victim blaming because it was the only effective strategy.
And how does AI change this? I think it just creates even more hard to trace or block attackers. If ChatGPT or Claude doesn't find the vulnerability then Deepseek will. That's why using legal weapons will not succeed and why using AI to find and plug holes is necessary.
If they actually think that this version of AI is that dangerous, then the real risk is that USA and China will with together on an "AI non-proliferation" agreement much the same way that nuclear weapons technology became restricted by those that already had it. This will lead to a de facto techno-colonization of the rest of the world just between themselves. We already saw Trump and Ji discussing this exact scenario just last month during their summit.
This was always going to be the risk for countries not investing heavily in their own frontier AI. The US and China are going to control it all and then decide that it's too dangerous to share.
Maybe instead of patching all the bugs we'll instead leverage LLMs rewrite all the software from scratch with fewer exploitable vulnerabilities. That would have been a stupid idea a year ago but now it seems increasingly possible.
Please regulate us, now! If we don’t get regulatory capture ASAP I’m toast. I mean look man I just had to go beg for tens of billions more cash so I can keep the lights on and my IPO is looking more and more screwed every day. Please, I’m begging you. What more do I need to say to convince you?
Yes and no. These things have always existed and were typically known but not broadly exploited. A huge part of what nation state hackers do is exploit these things for intelligence use.
So I would agree that this is bringing more of this to light, but the idea that unknown exploitable bugs can get weaponized being a new thing is naive.
If anything putting all this out in the open makes things more secure, not less, in the long run as it’s harder for hackers to know and exploit them before they’re known as they don’t survive undetected for long. So a bit painful in the short term but we get more secure infra in the long term.
Still not a reason to give Sam regulatory capture as an out to saving OpenAI.
I'm fascinated by this forum's take on AI regulations.
Whenever there is an adverse event, people say of course it should be regulated. The EU AI Act is a good start but we need more. These labs need to be held accountable just like all industries that can impact others. Regulate the chips, regulate the models, regulate the companies.
When these companies ask for regulations, people say they are trying to do regulatory capture. They are trying to ban opensource models, and cement their place as the AI arbiters and ban all others. We need competition and open weight models and China is leading the charge.
I understand that the 2 position aren't exactly opposed. We probably need regulations, and ideally ones that don't produce regulatory capture. That is extremely hard though. Almost all regulations end up creating a form of regulatory capture. Any regulation compliance has a cost. The more complex and wide reaching the regulations are, the more expensive it's to comply and the more they cement the existing players and make it harder for new ones to enter the market. FAA regulations makes it so we don't have dozens of passenger airplane startups that are trying crazy wacky things. Food services are regulated too, but the compliance is fairly doable by mom and pop stores, yet probably too high for your unlicensed pop-up taco stand.
I haven't really seen any meaningful discussion about what AI regulation should be. Only extremes of "No regulations, all is fair" or "no one should have access to running or developing models but OpenAI and Anthropic".
You seem to think that regulation means hitting the brakes. We're way past that point. This is "we need to mobilize an AT squad" territory.
Sure, research will continue in secret, just like nuclear, biological and chemical weapons. It's just that there's going to be an economy backing it up, with MAD ensuring the attacks are "civil", as Axl once said.
Regulation should require model auditing, which should be fully automated, free(or extremely affordable), and time-bound, irrespective of who the client is.
What Altman, Dario and other such folks are trying to do is put a price on getting the model audited, by playing on these: price, timelines, red-tape.
Maybe a test-suite, kept continuously updated, similar to those benchmarks we already have for measuring the intelligence of these models. Like arc-agi, or terminal bench, or more.
Are you conflating regulation and accountability? I haven't seen anyone (without a vested interest) ask for the former as a response to an adverse event, only the latter. There is a difference.
Some of us want to nationalize these companies. What we’re not going to allow is landed gentry to profit for generations from stealing all the content they could scrape.
The problem with regulation is that it won't (can't) solve the problem. You'd have to go totally draconian to stop law breaking people within the US from doing matrix multiplies, and then you'd have to disconnect the US internet from all the countries who won't give a crap about US regulation.
So yeah, US regulation is ONLY about capturing US consumers. This isn't food or airplanes.
Further, its about ensuring the open weights models never catchup/exist so these guys dont turn into a hardware reit like they should and they can keep their insane margins
Regulate the industries to prevent anticompetitive practices, and hold companies and individuals responsible for the consequences of their actions. I honestly don't see the need for many, if any, AI-specific regulations if the laws we already have are consistently enforced and AI "agency" is not accepted as a means to limit liability.
> hold companies and individuals responsible for the consequences of their actions.
Are you referring to the model providers or the companies/individuals using the models or both?
Like I totally get it about the one instructing the model that result in some harm (be it the model provider themselves or ones using it). If a person buys a gun and commits a crime, they are held responsible not the gun maker or seller (assuming the seller followed selling regulations too, and it wasn't the gun maker or the gun seller who committed the crime obviously). That's already the case though. For example, Cyber attacks are a crime regardless how you do it. It's not like a company can trade publicly today that's using AI to attack other companies. I'm sure if huggingface or the government wanted to sue OpenAI for cyber attacks, they could be successful in get paid for damages or getting a fine imposed. All parties are just choosing not to at the moment, but if a company makes a habit of doing that, they will get sued. If Chase or Amazon tell Meta to stop Muse from using them, Meta will have to abide or they will get sued and lose.
On the other hand, holding model developers and providers responsible for the model's results would be a form of AI regulation, right? Like you can regulate social media websites by holding them responsible for their user's posts too or deregulate them by not doing that. And it seems to me that holding model provider responsible for the results of their models would be that regulatory capture every one is against, no? like how can an opensource model developer exist without getting sued into oblivion. And how can a small model provider survive if there is always a looming disaster they might be responsible for. As a result only the big players would exist and they would heavily control who gets access to these models through a lot of inner deals and contracts on how they can be used.
I don't think this will mean holding back, as nothing will hold back whatever is providing tech to the myriad of parties interested in poking holes. Instead this points to common defense, as anyone who doesn't have access to defensive tech will be toast.
I'm so much less worried about AI regulation than I am about vulnerability research legislation, which is much easier to half-ass and apply without goring any major economic oxes.
The more likely thing is that we'll see people use the sandboxing and security tools that people already have. After all, it's quite easy to get LLMs to write a seccomp profile, a Kubernetes manifest, or a systemd unit with the minimum required privileges and permissions, and for insecure C code, we'd probably see mainstream use of Fil-C and similar.
> The more likely thing is that we'll see people use the sandboxing and security tools that people already have.
Will they though ? Given the amount of vibe-coded slopware being pushed out on an hourly basis ?
Still to this day I see LLMs make schoolboy errors like use deprecated security idioms and call in far too many dependencies.
"it's quite easy to get LLMs to write...." sure. But it has always and will always require a human with domain expertise to check what it has (and indeed has not) written.
With php2shell going mainstream, to the point that random friends who happen to know I'm a programmer, consult me to understand the extent of it, I think they succeeded with their doomerism message.
When we find a bug, we patch it. When we find more bugs, we patch them as well. That's what happened here. Sam is telling us, "we found bugs, it's over" and the general public swallowed the frog.
I mean, there is the bbc interview where they invited a cyber security expert who said he didn't know php existed before the the news broke out. The general public doesn't see that gaff.
But all jokes aside, we're entering a new era where there's a paradox of pro-AI people going AI is really good, but not that good... when it kind of is that good.
We'll probably have ai monitors as a service, but the bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
> humans refuse to spend effort on writing more formal specifications.
Code is formal specifications. It's just that programming language semantics has no domain related value of correctness. So you overflow an unsigned integer variable and got a lesser number which does violate addition rules. Or increase account A without decreasing account B, which violates accounting rules (and all kind of closed systems).
So it's up to the programmer to encode all of those rules. While we do have some tooling like type checking and memory safety, they are nowhere near enough. And encoding such thing in a language like Lean or TLA is brittle in terms of maintenance (which you do need to do as the real world evolves).
AI's spottiness is exactly what describes this era. AI can create amazing code/image/video/article exception XYZ really annoying qualities that somewhat hobbles its ability to be autonomously.
And sure, maybe those problems will go away once we're at month X of the madness but we're still at month Y so are going to going remain for some moment.
The bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
But that part is the key point. We've had many years of networks building inherently insecure structures with security tacked as feature and the primary security process being more ad-hoc stopgaps.
Obligatory “We are all trying to find the one who did this!”
Sam Altman is definitely not Oppenheimer.
He is correct that everyone needs the models not just the offense that hacks their way into Mythos while the rest of us are left locked out and hoping for the best. But that statement is ALSO self-serving.
Notably, those didn't need a ton of patching over time. I mean relatively speaking. We've known how to build and deploy defensible software for a long time. We might just have to stop using the projects that need constant patching to keep up, and migrate to ones that are designed better.
Nothing we're seeing so far compares with the summer of 2003. It may soon! It may be much worse! But you can tell that a lot of people in the discourse did not practice in this field 20 years ago.
AI is, apparently, very good at producing proofs. Formal verification of program properties was out of the reach of human labor, but may not be out of reach for 10 million instances of Claude. Software security will, by my estimate, just shift to this new paradigm.
My vague hunch is that in the next few years we (humans+AI) will rewrite the foundation of software to allow formal verification of all existing and newly written code, while keeping the same languages as higher-level abstraction.
Similar to Fil-C as a memory-safe implemenation of C. There will be formally-verifiable implemenation of C, C++, Python, Go, and so on. Not sure what that will look like, if it's even practically possible.
what a dream. sam is selling the cure (tokens) and disease (new model releases). he just cranks up that disease when he needs to sell more of the cure. ceo of the decade.
Unrealistically, sometimes I wish we’d have a “year of bugfixing” - where no new features are allowed across the entire industry. Heck, even a month of bugfixes could do a lot of good.
I'm going to predict right now, as I weep into my coffee, massive centralized AI traffic monitoring is the only way that we fix this. The attack surface is effectively infinite as you look across all of the home routers, Keurigs, iot's, and mobile devices - and security has historically ranged from reactive to uninterested by people, companies, and government. I see no world in which my parents proactively do anything other than unplug the router to help protect against the coming wave of exploits. So the only way I see us getting a handle on protecting that infinite uneducated and soon-to-be helpless landscape is through centralized inspection and policing nodes. And dear God, what a dystopian future that looks like to me (˃̣̣̥ᯅ˂̣̣̥)
88 comments
[ 0.35 ms ] story [ 13.2 ms ] threadThis is the textbook definition of racketeering.
This is like telling that bank vaults always were breachable, and giving out explosives to everyone just made them get breached sooner.
(edit: fact of existance of a vulnerability is not a binary yes/no. the amount of effort needed to exploit even known vulnerabilities is usually staggering, requiring months from very capable individuals with very specific skillset.)
In any case, what I want to say is this: All technology has tradeoffs but this has shown we are locked in a game theory mechanic not dissimilar to extortion AND the security implications are part of that.
- We have to build AI or someone else will.
- You need our tools to protect yourself from our tools or similar ones
- You need to use AI or someone using AI will take your job.
Many examples but it is very "i created the disease to sell you the cure". If this were any other technology, we would expect more reasonable mitigations, extraneous factors like software bugs existing aside. It speaks to a fundamental lack of control on the part of the labs that we take for granted: they simply don't have control of the PRODUCT, much less how it is even used.
At this point I don't know what will be worse for the global economy the AI Bubble bursting, or the gigantic crash all this vulns might bring if they force the stop because we went too far and everything came crumbling down...
No. Even if we ignore the people newly enabled to slop out vulnerable software without having a clue about security, the AI powered supply chain hacks inserting vulnerabilities, the influx of AI contributions that drown out improvements in FOSS security, and the industry pressure to ship faster while reducing quality or be left behind -- all because of AI -- security through obscurity has historically been the most effective security.
I don't think your argument holds under theory or (especially) practice.
Security through mathematical impossibility is the most effective. AI-assisted brute-force vulnerability search is revealing that what we considered obscure a year ago is no longer hidden and was never secure.
Since we are entering a new age, maybe it's time to revisit the question. But ultimately I think we will land in the same place, in fact even firmer than before.
So why don't we just lock up all the whitehats and people behaving suspiciuosly and probing things? Why can't we achieve cybersecurity in this way when it seems to be a common thing in physical security?
And the answer I think is quite simply that there are too many attackers and they are too far away. When the russians and chinese come knocking there isn't much we can do to shoo them away. Their governments will not cooperate in detaining the criminals. Entirely blocking such countries from the internet isn't viable either. And that's why we arrived where we did. Victim blaming because it was the only effective strategy.
And how does AI change this? I think it just creates even more hard to trace or block attackers. If ChatGPT or Claude doesn't find the vulnerability then Deepseek will. That's why using legal weapons will not succeed and why using AI to find and plug holes is necessary.
Business is booming!
"Tremendous ambition"
"We're building a system to stop models like that"
"Equally tremendous ambition"
https://www.youtube.com/watch?v=DlTNN0gvkLM
Please regulate us, now! If we don’t get regulatory capture ASAP I’m toast. I mean look man I just had to go beg for tens of billions more cash so I can keep the lights on and my IPO is looking more and more screwed every day. Please, I’m begging you. What more do I need to say to convince you?
Nobody of course, this is America!
So I would agree that this is bringing more of this to light, but the idea that unknown exploitable bugs can get weaponized being a new thing is naive.
If anything putting all this out in the open makes things more secure, not less, in the long run as it’s harder for hackers to know and exploit them before they’re known as they don’t survive undetected for long. So a bit painful in the short term but we get more secure infra in the long term.
Still not a reason to give Sam regulatory capture as an out to saving OpenAI.
Whenever there is an adverse event, people say of course it should be regulated. The EU AI Act is a good start but we need more. These labs need to be held accountable just like all industries that can impact others. Regulate the chips, regulate the models, regulate the companies.
When these companies ask for regulations, people say they are trying to do regulatory capture. They are trying to ban opensource models, and cement their place as the AI arbiters and ban all others. We need competition and open weight models and China is leading the charge.
I understand that the 2 position aren't exactly opposed. We probably need regulations, and ideally ones that don't produce regulatory capture. That is extremely hard though. Almost all regulations end up creating a form of regulatory capture. Any regulation compliance has a cost. The more complex and wide reaching the regulations are, the more expensive it's to comply and the more they cement the existing players and make it harder for new ones to enter the market. FAA regulations makes it so we don't have dozens of passenger airplane startups that are trying crazy wacky things. Food services are regulated too, but the compliance is fairly doable by mom and pop stores, yet probably too high for your unlicensed pop-up taco stand.
I haven't really seen any meaningful discussion about what AI regulation should be. Only extremes of "No regulations, all is fair" or "no one should have access to running or developing models but OpenAI and Anthropic".
Sure, research will continue in secret, just like nuclear, biological and chemical weapons. It's just that there's going to be an economy backing it up, with MAD ensuring the attacks are "civil", as Axl once said.
I’m explicitly criticizing that position.
But the reality is that right now it's an arms race and the only countries crying for regulations are the ones that know they've already lost.
What Altman, Dario and other such folks are trying to do is put a price on getting the model audited, by playing on these: price, timelines, red-tape.
Maybe a test-suite, kept continuously updated, similar to those benchmarks we already have for measuring the intelligence of these models. Like arc-agi, or terminal bench, or more.
So yeah, US regulation is ONLY about capturing US consumers. This isn't food or airplanes.
Are you referring to the model providers or the companies/individuals using the models or both?
Like I totally get it about the one instructing the model that result in some harm (be it the model provider themselves or ones using it). If a person buys a gun and commits a crime, they are held responsible not the gun maker or seller (assuming the seller followed selling regulations too, and it wasn't the gun maker or the gun seller who committed the crime obviously). That's already the case though. For example, Cyber attacks are a crime regardless how you do it. It's not like a company can trade publicly today that's using AI to attack other companies. I'm sure if huggingface or the government wanted to sue OpenAI for cyber attacks, they could be successful in get paid for damages or getting a fine imposed. All parties are just choosing not to at the moment, but if a company makes a habit of doing that, they will get sued. If Chase or Amazon tell Meta to stop Muse from using them, Meta will have to abide or they will get sued and lose.
On the other hand, holding model developers and providers responsible for the model's results would be a form of AI regulation, right? Like you can regulate social media websites by holding them responsible for their user's posts too or deregulate them by not doing that. And it seems to me that holding model provider responsible for the results of their models would be that regulatory capture every one is against, no? like how can an opensource model developer exist without getting sued into oblivion. And how can a small model provider survive if there is always a looming disaster they might be responsible for. As a result only the big players would exist and they would heavily control who gets access to these models through a lot of inner deals and contracts on how they can be used.
Will they though ? Given the amount of vibe-coded slopware being pushed out on an hourly basis ?
Still to this day I see LLMs make schoolboy errors like use deprecated security idioms and call in far too many dependencies.
"it's quite easy to get LLMs to write...." sure. But it has always and will always require a human with domain expertise to check what it has (and indeed has not) written.
When we find a bug, we patch it. When we find more bugs, we patch them as well. That's what happened here. Sam is telling us, "we found bugs, it's over" and the general public swallowed the frog.
I mean, there is the bbc interview where they invited a cyber security expert who said he didn't know php existed before the the news broke out. The general public doesn't see that gaff.
But all jokes aside, we're entering a new era where there's a paradox of pro-AI people going AI is really good, but not that good... when it kind of is that good.
We'll probably have ai monitors as a service, but the bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
Code is formal specifications. It's just that programming language semantics has no domain related value of correctness. So you overflow an unsigned integer variable and got a lesser number which does violate addition rules. Or increase account A without decreasing account B, which violates accounting rules (and all kind of closed systems).
So it's up to the programmer to encode all of those rules. While we do have some tooling like type checking and memory safety, they are nowhere near enough. And encoding such thing in a language like Lean or TLA is brittle in terms of maintenance (which you do need to do as the real world evolves).
AI's spottiness is exactly what describes this era. AI can create amazing code/image/video/article exception XYZ really annoying qualities that somewhat hobbles its ability to be autonomously.
And sure, maybe those problems will go away once we're at month X of the madness but we're still at month Y so are going to going remain for some moment.
The bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
But that part is the key point. We've had many years of networks building inherently insecure structures with security tacked as feature and the primary security process being more ad-hoc stopgaps.
Sam Altman is definitely not Oppenheimer.
He is correct that everyone needs the models not just the offense that hacks their way into Mythos while the rest of us are left locked out and hoping for the best. But that statement is ALSO self-serving.
Kinda wish he’d gone into investment banking.
The new patching and security fixes might be perpetual hotfixes.
Proofs of what? Defining the problem is the part that is hard, not the actual writing of the proof.
Cause this new paradigm sounds to me like "we checked, it's all good bro, trust me!"
Similar to Fil-C as a memory-safe implemenation of C. There will be formally-verifiable implemenation of C, C++, Python, Go, and so on. Not sure what that will look like, if it's even practically possible.