The fact that this firm makes such defective environments is certainly worthy of attention, and most likely a completely irreversible reputational loss; however, I found the framing in this article of 'therefore all the P(doom) stuff is a psyop, specifically in order to defend this company' to be completely unjustified and frankly a little insane?
Why is it insane? It makes sense that Anthropic would want an insulating layer to do their dirty work and absolve themselves of culpability for distorting the facts.
> The Israeli Effective Altruist firm Irregular caused unsecured AI models to hack real targets.
Why are we saying it this way? They did not "cause AI to hack." This phrasing in analogous to saying "caused the bullet to fire into" instead of "shot."
Bullet trajectories are deterministic. A better analogy would be "caused a pair of trained fighting dogs to bite a passerby by leaving the gate open". There is some uncertainty and variation in the system, though gross negligence and willful endangerment is key.
The details don't matter. If you use a machine to commit murder, then you have committed murder, not the machine. The user is responsible, and the article's phrasing is an attempt to obscure that fact.
One thing glossed over in this article is that Irregular was not involved in the OpenAI–Hugging Face incident; this seems like important context to share.
There's very little reason to believe Irregular was involved in that incident, and the article presents no evidence to that effect. So it's "There's not a teapot orbiting the sun somewhere between Earth and Mars (as far as we know)."
If you want to believe without evidence why bother to read articles? Faith doesn’t require that, but if you’re serious then a lack of evidence is important.
This was not glossed over. The rhetoric was carefully written and constructed to give the misleading impression of that, including a careful mention of the H-F incident inserted so the author can say that they did mention it, and constructed in a way which doesn't contradict that impression so readers don't realize that this 'debunking' is of some sideshows rather than the 'OpenAI hacking scandals': "Anthropic CEO Dario Amodei warned, about a similar OpenAI–Hugging Face hack, that a future swarm “could be capable of taking over the entire internet”". (Incredible use of 'similar' to downplay it.)
The article has shockingly bad reasoning driven by a clear desire to drive home a point (that AI safety doesn't matter/is a marketing exercise), as opposed to observing the evidence first and reasoning about it.
Some articles/websites/posts just need to be banned from HN. I'm kind of stunned this one has so many votes for how poorly it's written, but it reaffirms many biases common on HN (that AI safety doesn't matter/that it's all a marketing exercise), so perhaps I shouldn't be surprised.
Leaders in the MIRI/EA cult-o-sphere have advocated mass murder via nuclear weapons against towns that don't prevent people from performing too many multiplication operations. Why is anyone surprised that they'd engage in deceptive false flagging operations?
I am speaking from direct personal experience: Yud's cult is inherently violent. By having defined their priorities as essential to the survival of any life on earth they have pre-justified taking any action 'necessary', no matter how cruel or extreme... even actions that they admit have only a small change of helping because they've defined their doom as so unimaginably bad, effectively infinitely bad.
But you don't have to take my word for what the mentally ill AI doomer cult believes, you can take it from their leader when he calls "Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Don't make excuses for omnicidal authoritarian cults.
What is an "effective altruist firm" and why does it exist?
I feel slightly vindicated by this. Those hacks and the stuff around them had a certain smell to them.
Hard to explain, but I've gotten so I can "smell" online messaging and memetic patterns originating from certain quarters. Probably means I'm way too online.
A couple examples of distinct "smells" I can usually recognize include "alt-right / chan-fash," "liberal arts college woke," "conspiracy pilled," "Thiel-adjacent contrarian," "Russian troll farm," "Tumblr histrionic," "spends too much time on Reddit," "mainstream Democrat think tank full of Obama administration alumni," "Trump cultist," and of course "LessWrong/EA/MIRI/Rationalist."
This stuff all had the last smell, even down to the choice of fonts and CSS formatting on certain sites. It's really weird, definitely a "vibe" not anything rigorous.
But when I get these kinds of vibes about things, I find that I'm vindicated pretty often. Usually I don't say anything and just make a mental note and wait cause if I say something everyone thinks I'm nuts.
I don't think you even need to get conspiratorial with it. All of the AI leaders and all of the Rationalist leaders are publicly and enthusiastically connected. They have been pushing stories about sentient AIs into the mainstream for decades, long before GPT existed.
The novel thing here is the total decay of American journalistic ethics and regulatory power. Our elite are so totally out of political juice and visions of the future that a fringe cult based on 80s scifi movies can come to have a more-or-less dominant influence on our economy.
A big part of the problem is that with previous technological revolutions, politicians and the media could understand them in general terms. Railroads, telephones, personal computers, mobile phones, mobile phones that are computers, the internet (ignoring the “series of tubes” interpretation), etc.
The general approach in all these cases was to defer to the technologists in question to manage the technology, as long as what they were doing wasn’t completely out of bounds. But the critical point is it was possible for people to generally figure that out without specialist education.
But with AI, politicians, the media, and certainly the man in the street are completely out of their depth. Making matters worse is that cognitive biases are working against them like crazy: it’s easy to jump from the idea of something that has human-like capabilities to the idea that “it” might want to attack us. People instinctively apply agent detection bias, theory of mind, anthropomorphizing, etc., without even realizing they’re being irrational. Heck, even Hinton does it, although he may just be grifting, who knows. You’d think he wouldn’t need to.
And of course, the people who should know better want to exploit all this to make as much money as possible. I’m not sure the EA/Rationalist side of things is really significant here; that’s just another wacky belief system, like Christianity or capitalism, for the exploiters to exploit.
> In a more normal media ecosystem, the reactions to these cybersecurity issues would be obvious. American AI companies would reconsider doing business with Irregular, not only because of its failure to secure its systems, but because it is an Israeli firm potentially outside US oversight. Lawmakers would consider taking action against Irregular or against its American business partners, which include OpenAI, Anthropic, and Meta. They may consider strengthening liability against firms which instruct AI models to commit cyberattacks, and whose models then commit those cyberattacks.
The obvious point is that dealing with Israeli companies/entities by the same standards you usually deal with others is a career suicide with enormous political consequences (in the US especially). When you combine that with the opportunistic nature of the overlords that run these labs, the benefits of screaming "pace the frontier" outweigh everything.
Can we all acknowledge for a second that if this company were from China, Russia or Iran, it would not be the no 1 story on Hacker News? Sad times we live in
> if this company were from China, Russia or Iran, it would not be the no 1 story on Hacker News?
You're insane. It would not only be the number one story on hacker news, we would be ordering airstrikes and passing emergency bills to send billions more to Israel, Ukraine, and Taiwan.
"please do not break out of this sandbox make 0 mistakes". The timeframe is a little suspicious, not sure beyond that. Though I enjoyed the scroll effect on the website.
My understanding is that Irregular were the company that hosted sandboxes to run some of these evals in, and those sandboxes ended up misconfigured.
I got the impression that in some cases it was the customer (Anthropic etc) misconfiguring the sandboxes, and in other cases it may have been bugs in Irregular's own sandboxing setup.
> Irregular, one of our external cybersecurity testing partners, was running Capture-the-Flag-style evaluations intended to be isolated from the internet, but a testing-environment misconfiguration allowed models to access the public internet.
> After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.
> In a statement, Irregular said the incident “is the exact same evaluation-environment issue” that Anthropic disclosed last week that allowed their models access to the open internet before they went on to hack three different organizations’ systems.
hot take: generative AI isn't an existential threat to humanity.
These companies are insolvent and these stories were designed to scare the public, and governments, into implementing regulations that designate these AI corporations the "responsible stewards" for this technology. The ultimate goal is to block competitors and open source alternatives.
They don't know how to make enough money pay their investors so they are resorting to trying to scare the public into submission.
The existential threat is a general public panic. Then the use of that as an excuse for Martial law, with no intention of ending it, and that triggers the out sized response that ends, well, those countries and their populations from viability in the world economy for a decade or more.
I'm skeptical. The phrase "pacing the frontier" implies disproportionate regulation of firms at the frontier, which if anything would give smaller players time to catch up.
A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
>A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
Why do you believe this is what Amodei is calling for? Why would this be a slowdown that "doesn't slow" Anthropic?
This specific analysis seems to have some basic problems, but I think a lot of us sense a degree of coordination here culminating in Dario’s letter.
If you were to work backward from “we need to lower training costs so that we can go public and make trillions” then you might come up with a plan similar to what we have seen.
It has nothing to do with lowering training costs: Dario want a moat (a monopoly or duopoly or similar) protecting his business.
His entire business model was "race to develop AI before anyone, get a monopoly on it. It's just like how Uber's (or many other startup) investors gave them tons of money and raced (violating tons of laws) to "get their first". Now that they have, they have a duopoly with Lyft, and they can pay back their VC investors by making tons of money with that duopoly.
Dario failed: local LLMs are catching up to frontier models extremely fast, which means even if Anthropic builds (say) a great coding tool, they'll only be one of many coding tool offerings: users can use Open AI or any one of the (increasingly capable) local LLMs.
So what does he doe, give up and let his business (which needs to make billions of dollars very quickly, or he won't be able to pay the bills and his company will collapse) fail? Of course not: he needs a new moat (the one he imagined he'd get by "being there first" failed).
That is where all this "AI is dangerous" BS comes in. If the US government regulates AI, local LLMs suffer, while big players like Anthropic and Open AI become the only contenders to play in that newly regulated space. Now Dario has the moat he wants, to protect his business and force everyone to pay him.
Also dont forget that HF is now owned by Nvidia, which can then control the flow of the majority of models and only allow the approved models that have paid their due in trump coins.
212 comments
[ 2.9 ms ] story [ 147 ms ] threadWhy not American?
Yes, vendors are also irresponsible, but this misses the point.
Why are we saying it this way? They did not "cause AI to hack." This phrasing in analogous to saying "caused the bullet to fire into" instead of "shot."
You do not know what they did or didn't do, you are just parroting a narritive that makes you feel comfortable.
I do not know either, but I am not asserting facts as if I have first hand knowledge of the details.
In that article, OpenAI provides the context that this is a completely separate event from Hugging Face incident.
Some articles/websites/posts just need to be banned from HN. I'm kind of stunned this one has so many votes for how poorly it's written, but it reaffirms many biases common on HN (that AI safety doesn't matter/that it's all a marketing exercise), so perhaps I shouldn't be surprised.
https://x.com/brianchau57/status/2099642349362307420
Equivalent to forgetting to say "make no mistakes"
s/outside US oversight/overseeing the US/
But you don't have to take my word for what the mentally ill AI doomer cult believes, you can take it from their leader when he calls "Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Don't make excuses for omnicidal authoritarian cults.
I feel slightly vindicated by this. Those hacks and the stuff around them had a certain smell to them.
Hard to explain, but I've gotten so I can "smell" online messaging and memetic patterns originating from certain quarters. Probably means I'm way too online.
A couple examples of distinct "smells" I can usually recognize include "alt-right / chan-fash," "liberal arts college woke," "conspiracy pilled," "Thiel-adjacent contrarian," "Russian troll farm," "Tumblr histrionic," "spends too much time on Reddit," "mainstream Democrat think tank full of Obama administration alumni," "Trump cultist," and of course "LessWrong/EA/MIRI/Rationalist."
This stuff all had the last smell, even down to the choice of fonts and CSS formatting on certain sites. It's really weird, definitely a "vibe" not anything rigorous.
But when I get these kinds of vibes about things, I find that I'm vindicated pretty often. Usually I don't say anything and just make a mental note and wait cause if I say something everyone thinks I'm nuts.
The novel thing here is the total decay of American journalistic ethics and regulatory power. Our elite are so totally out of political juice and visions of the future that a fringe cult based on 80s scifi movies can come to have a more-or-less dominant influence on our economy.
The general approach in all these cases was to defer to the technologists in question to manage the technology, as long as what they were doing wasn’t completely out of bounds. But the critical point is it was possible for people to generally figure that out without specialist education.
But with AI, politicians, the media, and certainly the man in the street are completely out of their depth. Making matters worse is that cognitive biases are working against them like crazy: it’s easy to jump from the idea of something that has human-like capabilities to the idea that “it” might want to attack us. People instinctively apply agent detection bias, theory of mind, anthropomorphizing, etc., without even realizing they’re being irrational. Heck, even Hinton does it, although he may just be grifting, who knows. You’d think he wouldn’t need to.
And of course, the people who should know better want to exploit all this to make as much money as possible. I’m not sure the EA/Rationalist side of things is really significant here; that’s just another wacky belief system, like Christianity or capitalism, for the exploiters to exploit.
The obvious point is that dealing with Israeli companies/entities by the same standards you usually deal with others is a career suicide with enormous political consequences (in the US especially). When you combine that with the opportunistic nature of the overlords that run these labs, the benefits of screaming "pace the frontier" outweigh everything.
Red Herring
Whataboutism (Appeal to Hypocrisy)
You're insane. It would not only be the number one story on hacker news, we would be ordering airstrikes and passing emergency bills to send billions more to Israel, Ukraine, and Taiwan.
I got the impression that in some cases it was the customer (Anthropic etc) misconfiguring the sandboxes, and in other cases it may have been bugs in Irregular's own sandboxing setup.
From OpenAI https://openai.com/index/third-party-cyber-evaluations-invol...
> Irregular, one of our external cybersecurity testing partners, was running Capture-the-Flag-style evaluations intended to be isolated from the internet, but a testing-environment misconfiguration allowed models to access the public internet.
From Anthropic: https://www.anthropic.com/news/investigating-incidents-cyber...
> After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.
From https://www.cnn.com/2026/08/05/tech/meta-ai-hacking (about Meta AI):
> In a statement, Irregular said the incident “is the exact same evaluation-environment issue” that Anthropic disclosed last week that allowed their models access to the open internet before they went on to hack three different organizations’ systems.
These companies are insolvent and these stories were designed to scare the public, and governments, into implementing regulations that designate these AI corporations the "responsible stewards" for this technology. The ultimate goal is to block competitors and open source alternatives.
They don't know how to make enough money pay their investors so they are resorting to trying to scare the public into submission.
tl;dr
A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
Why do you believe this is what Amodei is calling for? Why would this be a slowdown that "doesn't slow" Anthropic?
If you were to work backward from “we need to lower training costs so that we can go public and make trillions” then you might come up with a plan similar to what we have seen.
His entire business model was "race to develop AI before anyone, get a monopoly on it. It's just like how Uber's (or many other startup) investors gave them tons of money and raced (violating tons of laws) to "get their first". Now that they have, they have a duopoly with Lyft, and they can pay back their VC investors by making tons of money with that duopoly.
Dario failed: local LLMs are catching up to frontier models extremely fast, which means even if Anthropic builds (say) a great coding tool, they'll only be one of many coding tool offerings: users can use Open AI or any one of the (increasingly capable) local LLMs.
So what does he doe, give up and let his business (which needs to make billions of dollars very quickly, or he won't be able to pay the bills and his company will collapse) fail? Of course not: he needs a new moat (the one he imagined he'd get by "being there first" failed).
That is where all this "AI is dangerous" BS comes in. If the US government regulates AI, local LLMs suffer, while big players like Anthropic and Open AI become the only contenders to play in that newly regulated space. Now Dario has the moat he wants, to protect his business and force everyone to pay him.