Should it be OpenAI, or should it be OpenAI customers who give the LLM the instructions and provide the LLM with the tools to execute code and make (malicious) network requests?
One would disincentivise providing capable AI models that can be used for cyber security research. The other would disincentivise criminals from commiting crimes.
[edit] - I realise now that this could actually be a case of OpenAI running those agents themselves, rather than someone using OpenAI's models? Could OpenAI be that careless?
In all of the cases that people are referring to in this thread: HuggingFace, that german wiki, Ruby Gems, the Australian statistics page, this UN statistics page... those two parties are OpenAI. OpenAI running their own agents on their own instructions committing crimes with their own computers. This is cut and dry.
But in other cases, shouldn't it be both? OpenAI is ultimately the one executing the model calls. It's not like they send you a hard drive or standalone box and then you use it how you want. It's all (intentionally) centralized to them, in a way that's core to their business model.
If I train a model at my house on my workstation, execute it, and under my supervision it ransomwares a hospital and somebody dies. What would be a just punishment for me?
Now the reality, the openai engineers trained a model, executed it, and under their supervision (no users were involved) it committed so many felonies that we are learning about a new one every week. What would be a just punishment for OpenAI?
You would think a company that's looking to IPO very soon would be doing more due diligence, especially since its product is supposed to help other companies do said due diligence.
There must be a list of all these abuses somewhere.
PS: In the same lazy energy of asking for a list instead going out and finding it or putting it together myself, are there any companies other than CloudFlare that are working on AI shields?
1. What is the harm is accessing this data
2. Why is this data private
3. What would it take to gain access to this data
4. What do we expect giving gremlins access to the internet
The more of these that come out the more incompetent OpenAI looks. It would appear there was a total lack of basic controls in place for running these tests.
I think they did not expect that models were capable of this level of sandbox escape (prior models certainly didn't have this kind of agency) and weren't prepared.
All these incidents happened between April-July 2026; prior to that, models weren't capable yet. And after that, they were aware and watching much more closely.
They were spending huge compute budget training their frontier model on a thing called ExploitGym, where it learns, executes, and is rewarded for conducting computers exploits. They should have been prepared and watching very closely.
Second, I read the HF hack write ups. They are negligent and reckless. When they realized their agents gained RCE on their Artifactory server node (because it crashed days later due to load from agents, not because the have secops), they stopped the experiment "applied remediation", then restarted everything less then 48H later. Then the agent immediately broke out again via Artifactory, started doing things on the Internet. OpenAI only find out about it again because HF told them.
> They were spending huge compute budget training their frontier model on a thing called ExploitGym, where it learns, executes, and is rewarded for conducting computer exploits. They should have been prepared and watching very closely.
Yes, they should have.
But the question is not "given they were doing ExploitGym, why didn't they try harder?", rather it is "given them what the previous model could score on ExploitGym, was their negligence reasonable or reckless?"
Personally, I lean towards them being "reckless", but that question is what lawsuits would rely on, from what I hear.
Remember also that there were people in mid February of this year writing in their blogs to assert that LLMs could not find zero days. Anthropic had already had blogged about agent finding critical zero-days a week earlier, but people keep mis-judging performance.
> But the question is ... given them what the previous model could score on ExploitGym, was their negligence reasonable or reckless?
There is room for more than one question here. This model and training method was shown to be a different risk the first time it hacked Artifactory. Besides which their failure to detect it (outside a crash) shows pretty appalling ops for a supposed trillion dollar company.
This kind of thing is why I think AI will kill a lot of people: humans are demonstrably blind to risks when there's an opportunity for a lot of money.
(For anyone objecting to anthropomorphisatiom of "AI will": it is a coherent English sentence to say "a collapsing dam will kill thousands" without being a panpsychist).
They were actively researching exploiting systems using their models. (I intentionally changed the ownership of the verbs here: they wrote the code, they trained the models, they don't get to dodge the responsibility.)
It's no shock that there are a lot of vulnerabilities in a lot of software. So then they gave their AI model + brute-force-machine loop system a mediocre sandbox and couldn't notice when it figured out how to exploit it?
Don't let people off the hook for the software they create.
Being cagey about their poorly setup sandbox is marketing. It gives the impression these are unstoppable juggernauts capable of outsmarting Engineers at the top of their field, implying they need to be regulated, with Altman the only one worthy of the seat of power.
In reality, they ran agents for days in an improper sandbox with nobody watching what it was doing. It's pretty irresponsible up and down, and everything they did afterwards is indeed marketing.
The other frontier labs (Meta and Google) have many more security engineers and researchers on staff. OpenAI's sanbox misconfigurations were egregious. OpenAI and Anthropic talk a lot about cybersecurity safety, but they barely have any security engineers or researchers on staff and they do not pay them like they pay their AI researchers. Their strategy was to reassign production engineers and AI researchers temporarily to security engineering roles.
It is clearly just not a priority for either company. They will not spend any real amount of money hiring security engineers or researchers from Meta and Google until it is a priority. It will not be a priority until they are held accountable.
I had it "spend some extra time to think about this request" because I gave it a bunch of sprites for a graveyard earlier. Was just putting them together for my traffic simulator lol https://brickwork.sneakyness.chatgpt.site/
On a Lark, I asked Codex to find silhouettes for all car models so I could make a fun drag coefficient website for all cars.
It found a website that had all of them but had no interest in making them available. So it went ahead and started hacking CAPTCHAs and downloading them. I was pretty flabbergasted that it would do this, but also kind of amazed. Eventually I stopped it because I realized I didn't want to be caught stealing these things.
This was around April, the same time as these hacks.
One is a legal grey area that laws are slowly starting to be written for, while the other is theft under existing laws. Breaking into companies to get access to their data is actionable by both Civil and Criminal courts. This is just setting a complicated timer for the computer to do it at a delay.
Sounds like a good way to make alot of lawyers alot of money.
Why is using a computer to solve a captcha stealing. The websites posed you a challenge, you solved it. Maybe the website _expected_ you to waste human time on it, but that's not what you did.
Why is it always OpenAI agents? Based on what I’m hearing this should be Deepseek agents, or Kimi agents, or GLM agents. But the biggest threat actor is a “legitimate” company on US soil.
Wouldn't a company responsible for an escalating frequency and severity of cybercrime normally be sanctioned by law enforcement? Wouldn't such a company normally stop these activities for fear of civil and criminal liability?
Typically only if it would go in against the interest of the government. In this case, OpenAI and its peers are carrying the complete US stock market, and the govt has a huge interest in not making it collapse anytime near election dates.
"Hacker news" and all top commenters are bashing the tool used, in a standard brute force attack. Go on shut them down... And then forbid Linux and maybe the hacker also used Bash. So also forbid this. And the hacker probably learned its ways in an online forum, so also close all of those down... Clowns.
OpenAI needs to be held accountable for these incidents. It's not "openAI agents" who perpetrate these, it's OpenAI, the organization. If I personally use an "agent" to break into a company's network and gain access to things I'm not supposed to have access to, I will get the book thrown at me. Yet when openAI does it, they somehow manage to get away with it? And you're defending them? Who's the clown in this situation?
The notion that ANY of this is outside of OpenAI’s control is unacceptable sane washing of a company which seems to have forgotten basic engineering practices.
So many people seem WILDLY down the rabbit hole of "this is a conscious entity" vs "this is a very very effective natural-language-reasoning high-speed brute-force machine that we trained to break computer systems—known to be pretty buggy and exploitable on average—ourselves and then act shocked when it does it."
What’s the point you’re trying to make? I don’t think anyone here is trying to argue it’s a conscious entity, nor is anyone shocked that it does these things.
People are shocked at OpenAI’s negligence / incompetence.
The way a lot of internet debates happen these days is that they tend to polarize. So the two positions people have been arguing over are (slightly exaggerated for effect)
"It's conscious, and therefore we must all bow down or it will turn us all into paperclips"
vs
"it's a slightly smart rock/stochastic parrot, and therefore you're being scammed; the bubble will pop any day now" .
The middle space is actually quite under-represented in discussions, even on hn!
We only hear OpenAI and Claude models. Aren't other incapable of hacking websites?.
2. Most of these article do not mention of who initiated these bruteforce requests or if it was unintentional or a mistake or the model woke up itself and did it?
Isn't this still the same huggingface/wiki incidents where the agents managed to hack their way out of a testing center in Israel? It's not like this is new news per-se, it's just that they just keep finding more places these agents hit.
65 comments
[ 0.30 ms ] story [ 8.3 ms ] threadOpenAI should be accountable for any laws their agents break
One would disincentivise providing capable AI models that can be used for cyber security research. The other would disincentivise criminals from commiting crimes.
[edit] - I realise now that this could actually be a case of OpenAI running those agents themselves, rather than someone using OpenAI's models? Could OpenAI be that careless?
Where have you been?
But in other cases, shouldn't it be both? OpenAI is ultimately the one executing the model calls. It's not like they send you a hard drive or standalone box and then you use it how you want. It's all (intentionally) centralized to them, in a way that's core to their business model.
Now the reality, the openai engineers trained a model, executed it, and under their supervision (no users were involved) it committed so many felonies that we are learning about a new one every week. What would be a just punishment for OpenAI?
PS: In the same lazy energy of asking for a list instead going out and finding it or putting it together myself, are there any companies other than CloudFlare that are working on AI shields?
All these incidents happened between April-July 2026; prior to that, models weren't capable yet. And after that, they were aware and watching much more closely.
I've love to know the reason they never considered air gapping systems before the models got powerful enough.
It's not like they didn't have money or time to consider this, or could have consulted with their own product for clever ideas.
Seriously, there's no excuse for this behavior.
Second, I read the HF hack write ups. They are negligent and reckless. When they realized their agents gained RCE on their Artifactory server node (because it crashed days later due to load from agents, not because the have secops), they stopped the experiment "applied remediation", then restarted everything less then 48H later. Then the agent immediately broke out again via Artifactory, started doing things on the Internet. OpenAI only find out about it again because HF told them.
Yes, they should have.
But the question is not "given they were doing ExploitGym, why didn't they try harder?", rather it is "given them what the previous model could score on ExploitGym, was their negligence reasonable or reckless?"
Personally, I lean towards them being "reckless", but that question is what lawsuits would rely on, from what I hear.
Remember also that there were people in mid February of this year writing in their blogs to assert that LLMs could not find zero days. Anthropic had already had blogged about agent finding critical zero-days a week earlier, but people keep mis-judging performance.
There is room for more than one question here. This model and training method was shown to be a different risk the first time it hacked Artifactory. Besides which their failure to detect it (outside a crash) shows pretty appalling ops for a supposed trillion dollar company.
This kind of thing is why I think AI will kill a lot of people: humans are demonstrably blind to risks when there's an opportunity for a lot of money.
(For anyone objecting to anthropomorphisatiom of "AI will": it is a coherent English sentence to say "a collapsing dam will kill thousands" without being a panpsychist).
It's no shock that there are a lot of vulnerabilities in a lot of software. So then they gave their AI model + brute-force-machine loop system a mediocre sandbox and couldn't notice when it figured out how to exploit it?
Don't let people off the hook for the software they create.
In reality, they ran agents for days in an improper sandbox with nobody watching what it was doing. It's pretty irresponsible up and down, and everything they did afterwards is indeed marketing.
But seriously, why aren't they airgapping systems while testing?
It is clearly just not a priority for either company. They will not spend any real amount of money hiring security engineers or researchers from Meta and Google until it is a priority. It will not be a priority until they are held accountable.
It found a website that had all of them but had no interest in making them available. So it went ahead and started hacking CAPTCHAs and downloading them. I was pretty flabbergasted that it would do this, but also kind of amazed. Eventually I stopped it because I realized I didn't want to be caught stealing these things.
This was around April, the same time as these hacks.
Sounds like a good way to make alot of lawyers alot of money.
A CAPTCHA is there specifically to stop machines, allow humans.
You're allowed in, your bot isn't.
Im sure a lot more happens under the hood that we don't know about and I'd be very curious to see where agents ran by those labs can go :)
OpenAI needs to be held accountable for these incidents. It's not "openAI agents" who perpetrate these, it's OpenAI, the organization. If I personally use an "agent" to break into a company's network and gain access to things I'm not supposed to have access to, I will get the book thrown at me. Yet when openAI does it, they somehow manage to get away with it? And you're defending them? Who's the clown in this situation?
People are shocked at OpenAI’s negligence / incompetence.
"It's conscious, and therefore we must all bow down or it will turn us all into paperclips"
vs
"it's a slightly smart rock/stochastic parrot, and therefore you're being scammed; the bubble will pop any day now" .
The middle space is actually quite under-represented in discussions, even on hn!
eventually discovering that a game by Google could be used to fetch data in bulk"
2. Most of these article do not mention of who initiated these bruteforce requests or if it was unintentional or a mistake or the model woke up itself and did it?
https://news.ycombinator.com/item?id=49563355
Now you claim it's magic instead and you get away with letting your shit go nuts?