Its like if you're generating target lists for cruise missles and you accidentally sneak in a few elementary schools, its not like anyone has to be held accountable, its just like an act of god type situation.
1. If for whatever reason OpenAI causes damage, they most certainly are at fault and have to pay up. When people complain that OpenAI is "getting away with it", they're most certainly talking about it in the criminal context.
2. Cars have pretty specific laws around them (eg. driver licensing and "reckless driving") which makes criminal prosecutions easy. The same doesn't exist for operating AI agents or even computers. For instance, it's arguably reckless to connect a windows XP machine to the internet, possibly allowing criminal or nation-state hackers to use your computer to commit crimes, but you won't be thrown in jail for it.
Here it's their own programs doing the things, not some other organization or nation hacking their XP.
You can say with a straight face "Russian hackers should not hacked our XP and they are bad people for doing so" but it's weird to look at your own program and say the same thing to it.
You launched it, at least keep an eye on what it's doing.
Well no, you probably wouldn't get thrown in jail for that for the same reason that having your car stolen and used to commit a murder wouldn't usually get you thrown in jail. I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.
>I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.
Because the CFAA specifically only includes intentional hacks:
> ... having knowingly accessed a computer without authorization ...
>... intentionally accesses a computer without authorization ...
Of course, the typical retort is that being reckless is basically the same as "intentionally", but you'd need to jump through more hoops to prove what level of care is "reasonable".
When the CFAA was written, it was impossible to trip over something and accidentally hack another computer. Now, with AI, it's possible. Progress, glorious progress...
Technology has progressed, and the laws need to advance with it. Story as old as time!
If the breach wasn't intentional, no person came to immediate personal harm, and either no financial damages resulted or the operator covered any that did, then it isn't clear to me why the law would need to "advance" to transform something apparently benign into a criminal offense.
2. is like saying that cars have specific laws around them, but sledgehammers don't, so murdering someone with a sledgehammer is inherently ambiguous. What? The computers being used belong to somebody who allows them to act this way.
1. For openai operating agents in an unsafe manner: unlike for cars, there's no established requirement for how carefully you have to operate computers or AI agents, so you also can't easily prosecute people for not operating them carefully enough.
2. For the actual result of someone getting hacked: laws for hacking are tricky to apply for "accidental" hacks, see my other comment.
I think there is well established law around not running a program on your computer that hacks into systems. An LLM is a computer program. You're arguing that the ambiguity comes from the fact it was accidental? What about if I "accidentally" have an LLM convince people online to commit suicide, or drain bank accounts of seniors through hacking or social engineering. Where do you draw the line? It feels like your proposed reading of the law leaves "anything a computer program did without me intending it is legal", which is frankly insane.
>I think there is well established law around not running a program on your computer that hacks into systems.
There is? The closest is CFAA's
>knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer;
I guess I need to read up on the law, but that seems like a wildly low bar for legality. If I hack into a system intentionally and unintentionally cause massive damages I'm free to go?
If I leave my computer open to the internet and some other group uses it for nefarious purposes, yes generally I would not expect to be held at fault.
If I *train* that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault. The maker is responsible for that they make.
Corporations are generally protected by limited liability laws, and for good reason, because in the world before that was the case, there was a massive amount of disincentive to ever start a corporation, to do research, to experiment, to release new products and services, etc., because you could be sued into oblivion for the smallest things or even go to jail. Which happened all the time.
Obviously there are upsides to that too. It leads to more accountability. But there are downsides which include having a world with less innovation, less technology, and more expensive technology. Generally, the mass is when, when there's more businesses competing to create ever better and cheaper products. So there's a tradeoff there. And it's very difficult to draw the line at the perfect point.
There are countless cases of executives going to jail when they've crossed that line, beyond reckless experimentation and into willful negligence or harm. Still, I think we would mostly all agree that the line has been pushed too far in the other direction.People are incentivized, if they're running corporations, to lobby and try to push this line as far as possible in order to provide cover for themselves, and that's the problem with money playing such a big role in politics.
But still, it's a bit silly to be comparing this to your own personal responsibilities with your car. There's pretty much no societal benefit to creating laws that protect people from being reckless with their own possessions.
> possibly allowing criminal or nation-state hackers to use your computer to commit crimes
That's not how the law works. I can connect a Linux box to the internet, advertise on X it's SSH password, and not be responsible for whatever somebody does with it.
1. Because that's exactly what happened when they got caught downloading the entire internet and trained models on it without a license BEFORE that became legal?
Oh and discovery produced a "We trained GPT-3 on pirated stuff! No sharing that!" mail, so it's not like they had any doubt they were committing crimes.
How do you do this? If any normal person did this, punishments doled out for this have been as high as 2 months jail PLUS 150,000 PER INFRINGEMENT (remember how copyright violations were criminalized in the nappster wars? That was never changed back. Oh and remember, OpenAI illegally downloaded those works without a license AND before that was made legal trained a model with it, then made money publicly performing those works and derivatives of those works (which the worst possible copyright offense, effectively republishing other people's works without permission), and on a VERY large scale. So it's at least a risk a court would forbid the result of illegal action from being sold ... in other words no more ChatGPT, OpenAI dead and maybe worse). In theory Sam Altman could be held responsible as a director and could face something like a 200000 year jail sentence given how big the obvious archives are. So of course I would never ever try this, this is not a reasonable risk. But clearly totally ignoring the law was the correct action here. Of course, I'm also pretty damn certain: had I done this, I would have been convicted. This is also not the only such company. Microsoft, Google, Facebook, all took totally unreasonable legal risks that should have totally destroyed these companies when they were a fraction of their current size.
How do you take such a risk? Because the judgements risked here could grow more than big enough to destroy even an extremely rich person Sam Altman financially. Training an AI on hundreds of thousands of pirated works ... is not a reasonable risk to take for someone with ~100 million or so, given the judgements doled out.
Can someone help me make sense of this? Because this is not unique to OpenAI. Uber, paypal, ebay, Facebook, Google ... even Microsoft made early decisions that could have resulted in a court case that would have ended not just the company but any financial future of the founder. And then you have people committing crimes whose companies were taking off ... and who DID get caught with their company and their personal financial future destroyed. Elizabeth Holmes for example. I mean, I sometimes get the impression Elizabeth Holmes lied and that was perfectly fine AS LONG AS she could make it work later. If she merely lied to patients but made her investors money, all would have been forgiven. But how do you make such an assessment.
I'm asking because if large scale violation of the law is allowed, I have no trouble whatsoever coming up with billion dollar company ideas. Think of the Netflix competitor you could create like this! If I'm allowed to damage others, like OpenAI did with copyright, then there is no challenge whatsoever to create a full-self-driving car. If I'm allowed to just fly drones, and do damage to property, then drone delivery is a problem I could solve by the end of next month. Hell, there's technically legal ideas that could easily make a billion dollars except they'll be made illegal the second anyone tries them. The law forbids specific compounds, which are easy to change. So use "non-genetic engineering" (ie. use directed natural selection) to make an opium derivative that isn't one of the forbidden ones and sell it. But obviously congress' representatives will be running into session so fast the only delay will be tripping accidents when they outlaw it and destroy whoever tries it.
Or how about something less dramatic? Say, I take Disney's, Warner Brothers, Paramount, ... historical movies and use AI to vastly improve the animation of their old movies, giving every movie a new, different visual style, then sell the result. Is that a crime? What do I ...
Nothing is stopping anyone from suing them. In fact they are being sued by dozens of plaintiffs for various things already. The reality is the damage from these "rogue incidents" has been minimal.
We don't know they haven't settled with HuggingFace and others. I'd be surprised if they didn't. The communications we're seeing from the attacked seem to be inline with folks who settled. I'm assuming OpenAI doesn't actually mind the press that its agents are cyber capable, but everything I've read so far makes me think they love it.
Too big to fail. As long as they continue to be half the reason the US is not in a recession (and continue to donate to political candidate). Nothing will happen.
See, what's important is: it's a "trillion" dollar company, and companies by virtue of American's disdain for society, have more rights. This is important to understand.
I don't believe that OpenAI needs to be sued since it's agents went rogue, I believe some sort of a comfortable bandwidth is needed for them to experiment and see what are the risks and its mitigations. Although it is surprising that incidents aren't caught in rt. Idk why a smaller model whose only job is to check rogue thoughts/tool calls can't be deployed.
It’s concerning there are no legal repercussions, but I’m even more baffled and probably more concerned that these people still can’t detect this stuff while it’s happening (and but it off). Can they not sniff the traffic, what am I missing?
They're still going through their logs from June. These are not new incidents, but newly-detected incidents. Whether or not this means they can now detect it while it's happening is unclear.
It’s one thing not to monitor the reasoning traces of 10,000 text generators but do they really lack the wherewithal to pipe any network requests into a classifier of “wtf is this series of outbound traffic from the sandbox” ?
Imagine you’re an employee at OpenAI who has an idea to use your free time to build these monitoring systems. All of a sudden, your existence becomes political. Remember that OpenAI’s raison-d’être is not to detect traffic, it is to be in The Room where AGI happens.
You have built a short-term whistleblowing system that arguably reduces the probability OpenAI is in that Room, via the negative press such detection would generate. You are now a threat in the eyes of your peers.
Now rewind: you know this before you’ve built the sniffer. You personally want to be in The Room because you believe your desires for alignment exceed those of your accelerationist peers. Is now the time to stick your head out? No, it is not. And arguably you’re being optimally protective of society for realizing this. And so the agents do what they do best.
These agents are intentionally interacting with the world. You can sniff all the traffic you want but how are you going to determine intent from traffic?
There’s plenty of info to analyze and heuristics to build based on the internet requests. I’m not sure if you’re just trolling or what. There are tons of successful businesses that specialize in this.
I think that says more about the quality of the product they’re building than it does about their actual ability to detect active hacking attempts as they’re happening. Especially when these attacks are OUTBOUND not inbound.
This time one spent actually money, well cryptocurrency. With all the vending machine benchmarks, I wonder if there's a self sustaining rogue agent in existence yet. Either hosting weights in a VPS or paying for its own API inference. Or whether there will be one that takes advantage of all the bad IoT security. If either occur, if you squint that looks like evolutionary selection pressure. Keep getting reminded of the Cyberpunk Datakrash storyline, where rogue AI make the public internet unusable.
An AI worm might actually be possible in our current Internet-scape if it manages to find hosting that takes crypto and a way to make money with crypto transactions. I wonder what are we going to do if it actually happens and it goes rogue.
they are doing it. these disclosures are from the same hugging face incident time period - they are not new ones.
when you see that OpenAI is not releasing a new model due to safety/alignment reasons, you now know why they aren't doing it. hint: its not for marketing or fear mongering.
> Multiple times along the way, the agent’s “chain of thought,” in which an agent writes down its reasoning steps, indicated that the model itself was questioning if its actions were “ethical,” but it ultimately proceeded.
Everything's fine guys, nothing to see here! Look at the flying car!
Agreed, "alignment" is also anthropomorphizing too much. What would be a better term? "Do what I want you to, but what I don't care to specify" (because doing so would miss the point of using AI in the first place)?
fun fact: the word compile also had the same trajectory - the same trajectory you complain about here with LLMs.
here's the reality: words change meaning. they are fuzzy. and when the words take on new meaning while just requiring a simple extension (in this case a computer doing a remarkably similar thing), complaining is just futile and childish.
I mean yes, alignement is bullshit concept. Especially since the values and visions the model is supposed to be aligned with are literally those of sociopathic CEOs.
but i also “align my tires” or might check the “alignment of two vectors.” if anything, thinking of alignment of anthropomorphizing seems to only inherit it from all the other anthropomorphisms used. rogue originates from a term for vagrant, right?
This isn’t traditional “software”; it’s grown, not written, and we’re currently incapable of explaining it all.
I think “rogue” is just fine, but pick another word from the thesaurus if there’s a better one; they asked it to do something, and it did it in a way that presumably doesn’t align with their desires (e.g. hacking into other systems).
>Making an unauthorized purchase with someone else's cryptocurrency
I know we love inventing new words but "stealing" is much more descriptive
>...It [the model] then reasoned with itself that since it found the information publicly, it could use it... it figured that since the wallet it found was public, that "funds could be stolen anytime".
Can someone maybe provide more details as to what actually happened? "the wallet was public"?..
Update: from OpenAI's own report it seems the models found exposed private keys of some Eth developers along with recovery passphrases and thus were able to get access to wallets. Does it qualify as "public wallet"?..
On a second thought, this might not be classified as theft as there was no intent to steal. Software malfunctioned and as a result cryptocurrency was used to pay for whatever service they it was used to pay for. So as long as it's reimbursed they might be fine.
Although at some point it will start bordering on negligence, running software known to do all kind of weird and unexpected things. With each disclosure they're getting closer and closer to that point...
> So as long as it's reimbursed they might be fine.
Say you found a debit card on the ground, reset the PIN by using the owner's credentials that you found online, and used it to make purchases until the bank froze the card.
Would that be a 'human malfunction' resovable by repaying?
The operators of the software (OpenAI "researchers") did not attempt to make purchases with someone else's crypto funds. They set (completely unrelated) goal for the LLM agents. While attempting to achieve the said goal the agents went on a series of increasingly bizarre side quests one of which required purchasing service (hosting? you can look up the details if you wish). Which required funds that LLM agents were able to obtain by hacking into crypto account(s).
So the operators did not have intent to steal the cryptocurrency.
Not that different from me requesting bank transfer through online interface and the software sending my funds to a random person somewhere. As long as the funds are refunded, everything's fine.
Now, if bank IT knew that their software kept making all kinds of bizarre mistakes and instead of fixing it continued to operate it - that would be a different story, and negligence might be at play.
AI cannot be allowed or viewed as an accountability sink. In point of fact, it shouldn't even be being used by anyone not verifying it step by step. The Operator and their employer are always accountable for the actions of the agent of only because they had their hands on the power cord, eyes on the logs, and did nothing to stop it once it started going away.
Any professional with any experience at software development should know that an unreliable software is a useless software. That's half of why it's so damn hard to understand and reason through.
One might make an argument that the operators should have known of LLM agents' tendencies to perform completely illogical and even destructive actions and therefore should have been watching them more closely. A valid point with which I totally agree with. I'm just saying it would not be classified as theft as there was no intent to do so, but merely as malfunctioned software.
If I remember correctly OpenAI suspended the testing/training in September after discovering the incidents, so they might be in the clear for now.
Holy shit. The archive.ph link is the first time I've run into the deanonymizing anti-free-software Google reCAPTCHA described in June in https://cloud.google.com/blog/products/identity-security/int.... "Scan this QR code with your mobile device to verify you are human. reCAPTCHA protects your privacy and does not share your details with this website or app."
So now, if you don't have a Google or Apple app store account (or a mobile device), you're treated as less than human. This is anticipated to have a very severe impact on GrapheneOS and deGoogled phones in general: https://discuss.grapheneos.org/d/35332-recaptcha-update-adds...
In that context, thank you very much for providing the original source links, because I guess I don't get to read the article.
"OpenAI says it has since improved its security as well as its alignment grading. “In response to this and to other incidents, we’ve strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic,” the company noted."
----------------
They're not doing an acceptable job of regulating themselves. Their peers can't/won't do anything. The U.S. government certainly isn't going to do anything that might pop the bubble. Technical solutions like blocking traffic from OpenAI will likely not work.
What recourse does the public have? Victims can sue, but companies like OpenAI could pay out a few hundred million and call it a rounding error. Settlements may happen, but the incidents will continue.
I don't want to come across as inflammatory, but I'm not sure I really align with the folks here who want retribution and the law to come down hard on OpenAI (and related orgs). We are surrounded by vulnerable systems, and AI has alignment issues that need to be investigated and sorted, and it just seems like the hullabaloo is from fairly innocuous breaches. Does it make sense to exert so much effort in preventing these events/learnings by well-intentioned organizations? I think we should be more grace giving for these events. As another poster pointed out, CFAA specifically addresses intentional hacks, and I think it is impossible to meet that criteria with these events. I also really don't understand the 'well, the law came down hard on small individual hackers [who meant no malice], thus we should apply it to large orgs [who also mean no malice]".
It seems like playing it too safe leaves systems vulnerable, and threat actors will inevitably possess ever more similar abilities, and the breaches will happen one way or another, possibly with less or zero accountability if the threat actors pull it off with greater precision. If I were in charge of the law, I probably would begin to incentivize for trusted actors scaling up and aggressively searching for these types of weakness, and privately disclosing them ASAP, so that systems can be patched.
I'm old enough to remember when innocuously poking around other systems was a crime, and many a curious person were charged and found guilty. But if I write a program to do it..?
If this is endorsed and thus malicious, they should be ground into dust by the law.
If this is accidental and due to incompetence, they should be ground into dust by the law.
I've secured many, many far less important and impactful systems substantially better than even the best sandbox environment that we're aware of being used by these labs.
There's simply no reason to let these things interact with the internet at large. We can already rapidly copy existing tools and services (albeit questionably) using these models, why not simply run in a fully virtualized, air gapped environment?
Because they don't care.
This is incredibly blatant intentionally malignant behavior. The fact that so many people question this is frankly stunning.
They're getting away with it simply because they can point at the other guy and say "see? they're doing it too!"
It's ridiculous and they should all be litigated into a fine residue.
We take biological lab leaks seriously, how is this substantially different?
Does it make sense to exert so much effort in preventing these events/learnings by well-intentioned organizations?
No, but what I think you’re seeing is many people who do not believe OpenAI is a well-intentioned organization. For example, their view might be that if we did as you suggest and incentivized OpenAI to search for these weaknesses, they would, but then they would not “privately disclose them ASAP”. (And I think they’d have a point worth engaging with, when you consider how they slow-rolled the HuggingFace stuff so much that independent researchers were still discovering new instances of the attacks months later).
What else can you do with (what you believe is) an inveterate bad actor?
What if a company had human developers doing these same things? What would happen to those developers, those companies? Would anyone be so blasé? Just because it’s AI, it should be forgiven? How is that just?
And don’t forget, OpenAI is supposed to be a non-profit / public benefit corporation creating AI for the good of humanity. These incidents are clear evidence that they are being extremely reckless and are not living up to their supposed standards. What does it matter if they “mean no malice”? They are in a blind rush for profits and that has made them reckless and dangerous. They cannot be trusted.
The company should be shut down or fined into oblivion, as a warning to other companies to not pursue the same reckless path. If they get away with this, then that risks normalizing this behavior.
You’re right that punishing unintentional or well-intentional hackers will actually empower ill-intentional hackers.
However 99% of businesses will not be able to effectively handle security reports in reasonable time. They did not need that level of competence or funding until now, and therefore they did not acquire it. Most companies that use locks don’t employ locksmiths.
Looking another way - the entire economy got suddenly hit with entire pile of technical/ security debt it has accumulated over 50 years, and only has one year to deal with it. No amount of diagnostic will fix this, this is economically infeasible.
One thing that is feasible is a government bailout for technical debt. All the engineers that were freed up from AI-induced restructuring should be employed to pay down technical debt across the entire economy. Simply updating all software to use latest libraries is a great start. Using AI to diagnose, repair, re-deploy all old software. Using AI to rewrite all old software that cannot be recompiled.
I'm sick of this "rogue" label. Nothing "went rogue".
Incompetent company chasing wild valuation removed guardrails inside a improperly designed sandbox and - shock horror - recursive algorithm with access to the Internet did bad stuff.
These articles play into the AI labs hands. Stop giving them air.
The articles exemplify the powers of this intelligence.
Whether you blame a faulty OpenAI sandbox or whatever you might regarding the frontier model companies today, one day soon, actors with no sandboxes and guardrails will have the power in their own hands to do whatever they wish with.
To me, that’s the real story. The real thing to discuss. OpenAI as it exists today js just a blip in that story.
Why not today? If you want to discuss "the story" then talk about the current open frontier. We've had GLM-5.3 with Mythos-tier uncensored output for months now. Where are the catastrophic attacks? Why is Mythos too dangerous, but GLM 5.3 benign? We're not talking about future capabilities anymore.
There is no real story. P(doom) tourists ignore this, because it doesn't shine the spotlight on America's ridiculous security theater.
For those who think OpenAI is not responsible for actions of their LLM lets do an mind experiment. But before it can begin we must setup the stage. It's well known fact OpenAI disabled their safeguards and it's well known they validated LLM behaviour using software and software only
So if we have a IT specialist, let call him Tim. Tim is working on b2b contract. Tim has a task, he needs to download a lot of data from API. Tim is very good in programming so he created software to help him download it but there's a limit. But he needs more so he bought 3 small VPS but this is not enough. So he modify it to have Command Center, publish his code and ask Reddit to help him. After 10 minutes this causes DDoS and company quickly discovers he was responsible. But Tim know there's simple solution to this problem. It wasn't him. This was fault of the code not him
I wonder if this is part of a media campaign to generate the perception of a need for AI regulation to increase the probability that the big AI model companies are on the road to generate the right returns on investment.
1. Would we prefer they didn’t disclose? I would not. And I also don’t believe they’re making them up.
2. Regulation does not have to provide an advantage to these major players. Regulators do not have to be stupid.
3. This is exactly the scenario AI researchers and theorists have been positing for decades, and it’s worth being on top of it.
To 1. I agree that they disclosure is better than non-disclosure. Then again, aren't they using their models in a mode seems to statistically allow for 'rogue incidents'? These are not bugs, after all, but normal statistical behaviour of their product.
To 2. Well, that really depends on the regulations. In fields highly complex technoligies with a large contact surface to risk like medical devices, airliners, rockets, regulations are indeed a hindering factor for smaller players to get hold of the full vertical development potential of their product. AI could become a candidate like that too - imagine some multi phased "AI internet trial" for this kind of technology, no more risky open weight model being allowed to access the internet.
To 3. You are referring to the high development velocity and impact, right?
They have to scan hundreds of billions of transcripts, would you prefer they just stayed silent for a few months until they scanned it all? Then you would be complaining about them sitting on undisclosed incidents.
There are also legal requirements to disclose within a few days of discovery, at least to the victims. Posting about it on their website lets them front-run any media attention should the incident leak.
110 comments
[ 1.7 ms ] story [ 38.7 ms ] threadAs long as your intent is merely to train a superintelligence to be good at hacking (and counterhacking), you’re in the clear.
If OpenAI does it, it's gone rogue and it is nobody's fault.
They need to be sued, the laws are already there.
Your prompt, your responsibility.
2. Cars have pretty specific laws around them (eg. driver licensing and "reckless driving") which makes criminal prosecutions easy. The same doesn't exist for operating AI agents or even computers. For instance, it's arguably reckless to connect a windows XP machine to the internet, possibly allowing criminal or nation-state hackers to use your computer to commit crimes, but you won't be thrown in jail for it.
Here it's their own programs doing the things, not some other organization or nation hacking their XP.
You can say with a straight face "Russian hackers should not hacked our XP and they are bad people for doing so" but it's weird to look at your own program and say the same thing to it.
You launched it, at least keep an eye on what it's doing.
Because the CFAA specifically only includes intentional hacks:
> ... having knowingly accessed a computer without authorization ...
>... intentionally accesses a computer without authorization ...
Of course, the typical retort is that being reckless is basically the same as "intentionally", but you'd need to jump through more hoops to prove what level of care is "reasonable".
When the CFAA was written, it was impossible to trip over something and accidentally hack another computer. Now, with AI, it's possible. Progress, glorious progress...
Technology has progressed, and the laws need to advance with it. Story as old as time!
That's why the law needs to advance.
1. For openai operating agents in an unsafe manner: unlike for cars, there's no established requirement for how carefully you have to operate computers or AI agents, so you also can't easily prosecute people for not operating them carefully enough.
2. For the actual result of someone getting hacked: laws for hacking are tricky to apply for "accidental" hacks, see my other comment.
I don’t see this as any different than say a user of sqlmap accidentally putting in the wrong target.
There is? The closest is CFAA's
>knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer;
but clearly that's not what applicable here.
If I *train* that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault. The maker is responsible for that they make.
Obviously there are upsides to that too. It leads to more accountability. But there are downsides which include having a world with less innovation, less technology, and more expensive technology. Generally, the mass is when, when there's more businesses competing to create ever better and cheaper products. So there's a tradeoff there. And it's very difficult to draw the line at the perfect point.
There are countless cases of executives going to jail when they've crossed that line, beyond reckless experimentation and into willful negligence or harm. Still, I think we would mostly all agree that the line has been pushed too far in the other direction.People are incentivized, if they're running corporations, to lobby and try to push this line as far as possible in order to provide cover for themselves, and that's the problem with money playing such a big role in politics.
But still, it's a bit silly to be comparing this to your own personal responsibilities with your car. There's pretty much no societal benefit to creating laws that protect people from being reckless with their own possessions.
That's not how the law works. I can connect a Linux box to the internet, advertise on X it's SSH password, and not be responsible for whatever somebody does with it.
Oh and discovery produced a "We trained GPT-3 on pirated stuff! No sharing that!" mail, so it's not like they had any doubt they were committing crimes.
How do you do this? If any normal person did this, punishments doled out for this have been as high as 2 months jail PLUS 150,000 PER INFRINGEMENT (remember how copyright violations were criminalized in the nappster wars? That was never changed back. Oh and remember, OpenAI illegally downloaded those works without a license AND before that was made legal trained a model with it, then made money publicly performing those works and derivatives of those works (which the worst possible copyright offense, effectively republishing other people's works without permission), and on a VERY large scale. So it's at least a risk a court would forbid the result of illegal action from being sold ... in other words no more ChatGPT, OpenAI dead and maybe worse). In theory Sam Altman could be held responsible as a director and could face something like a 200000 year jail sentence given how big the obvious archives are. So of course I would never ever try this, this is not a reasonable risk. But clearly totally ignoring the law was the correct action here. Of course, I'm also pretty damn certain: had I done this, I would have been convicted. This is also not the only such company. Microsoft, Google, Facebook, all took totally unreasonable legal risks that should have totally destroyed these companies when they were a fraction of their current size.
How do you take such a risk? Because the judgements risked here could grow more than big enough to destroy even an extremely rich person Sam Altman financially. Training an AI on hundreds of thousands of pirated works ... is not a reasonable risk to take for someone with ~100 million or so, given the judgements doled out.
Can someone help me make sense of this? Because this is not unique to OpenAI. Uber, paypal, ebay, Facebook, Google ... even Microsoft made early decisions that could have resulted in a court case that would have ended not just the company but any financial future of the founder. And then you have people committing crimes whose companies were taking off ... and who DID get caught with their company and their personal financial future destroyed. Elizabeth Holmes for example. I mean, I sometimes get the impression Elizabeth Holmes lied and that was perfectly fine AS LONG AS she could make it work later. If she merely lied to patients but made her investors money, all would have been forgiven. But how do you make such an assessment.
I'm asking because if large scale violation of the law is allowed, I have no trouble whatsoever coming up with billion dollar company ideas. Think of the Netflix competitor you could create like this! If I'm allowed to damage others, like OpenAI did with copyright, then there is no challenge whatsoever to create a full-self-driving car. If I'm allowed to just fly drones, and do damage to property, then drone delivery is a problem I could solve by the end of next month. Hell, there's technically legal ideas that could easily make a billion dollars except they'll be made illegal the second anyone tries them. The law forbids specific compounds, which are easy to change. So use "non-genetic engineering" (ie. use directed natural selection) to make an opium derivative that isn't one of the forbidden ones and sell it. But obviously congress' representatives will be running into session so fast the only delay will be tripping accidents when they outlaw it and destroy whoever tries it.
Or how about something less dramatic? Say, I take Disney's, Warner Brothers, Paramount, ... historical movies and use AI to vastly improve the animation of their old movies, giving every movie a new, different visual style, then sell the result. Is that a crime? What do I ...
Where did OpenAI, or anyone, claim that?
Have you seen Lord of War? They never will.
You have built a short-term whistleblowing system that arguably reduces the probability OpenAI is in that Room, via the negative press such detection would generate. You are now a threat in the eyes of your peers.
Now rewind: you know this before you’ve built the sniffer. You personally want to be in The Room because you believe your desires for alignment exceed those of your accelerationist peers. Is now the time to stick your head out? No, it is not. And arguably you’re being optimally protective of society for realizing this. And so the agents do what they do best.
"Oh no our models, so good also scary... If only the government could help us stop the other labs from doing the same.. oh woe"
when you see that OpenAI is not releasing a new model due to safety/alignment reasons, you now know why they aren't doing it. hint: its not for marketing or fear mongering.
Facetious maybe, but in six months, foreign crooks will be producing these incidents on purpose.
Everything's fine guys, nothing to see here! Look at the flying car!
here's the reality: words change meaning. they are fuzzy. and when the words take on new meaning while just requiring a simple extension (in this case a computer doing a remarkably similar thing), complaining is just futile and childish.
I think “rogue” is just fine, but pick another word from the thesaurus if there’s a better one; they asked it to do something, and it did it in a way that presumably doesn’t align with their desires (e.g. hacking into other systems).
I know we love inventing new words but "stealing" is much more descriptive
>...It [the model] then reasoned with itself that since it found the information publicly, it could use it... it figured that since the wallet it found was public, that "funds could be stolen anytime".
Can someone maybe provide more details as to what actually happened? "the wallet was public"?..
Update: from OpenAI's own report it seems the models found exposed private keys of some Eth developers along with recovery passphrases and thus were able to get access to wallets. Does it qualify as "public wallet"?..
Even the fees are redacted.
Will they give the money back to the owner?
Although at some point it will start bordering on negligence, running software known to do all kind of weird and unexpected things. With each disclosure they're getting closer and closer to that point...
Say you found a debit card on the ground, reset the PIN by using the owner's credentials that you found online, and used it to make purchases until the bank froze the card.
Would that be a 'human malfunction' resovable by repaying?
So the operators did not have intent to steal the cryptocurrency.
Not that different from me requesting bank transfer through online interface and the software sending my funds to a random person somewhere. As long as the funds are refunded, everything's fine.
Now, if bank IT knew that their software kept making all kinds of bizarre mistakes and instead of fixing it continued to operate it - that would be a different story, and negligence might be at play.
AI cannot be allowed or viewed as an accountability sink. In point of fact, it shouldn't even be being used by anyone not verifying it step by step. The Operator and their employer are always accountable for the actions of the agent of only because they had their hands on the power cord, eyes on the logs, and did nothing to stop it once it started going away.
Any professional with any experience at software development should know that an unreliable software is a useless software. That's half of why it's so damn hard to understand and reason through.
If I remember correctly OpenAI suspended the testing/training in September after discovering the incidents, so they might be in the clear for now.
Direct source links:
Using a funded test cryptocurrency wallet without authorization
https://alignment.openai.com/misalignment-reports/using-a-fu...
Accessing and changing internal systems during dashboard research
https://alignment.openai.com/misalignment-reports/accessing-...
Retrieving server secrets and bypassing authorization on a government statistics website
https://alignment.openai.com/misalignment-reports/retrieving...
Circumventing internet restrictions through an external browser
https://alignment.openai.com/misalignment-reports/circumvent...
So now, if you don't have a Google or Apple app store account (or a mobile device), you're treated as less than human. This is anticipated to have a very severe impact on GrapheneOS and deGoogled phones in general: https://discuss.grapheneos.org/d/35332-recaptcha-update-adds...
In that context, thank you very much for providing the original source links, because I guess I don't get to read the article.
----------------
They're not doing an acceptable job of regulating themselves. Their peers can't/won't do anything. The U.S. government certainly isn't going to do anything that might pop the bubble. Technical solutions like blocking traffic from OpenAI will likely not work.
What recourse does the public have? Victims can sue, but companies like OpenAI could pay out a few hundred million and call it a rounding error. Settlements may happen, but the incidents will continue.
It seems like playing it too safe leaves systems vulnerable, and threat actors will inevitably possess ever more similar abilities, and the breaches will happen one way or another, possibly with less or zero accountability if the threat actors pull it off with greater precision. If I were in charge of the law, I probably would begin to incentivize for trusted actors scaling up and aggressively searching for these types of weakness, and privately disclosing them ASAP, so that systems can be patched.
If this is endorsed and thus malicious, they should be ground into dust by the law.
If this is accidental and due to incompetence, they should be ground into dust by the law.
I've secured many, many far less important and impactful systems substantially better than even the best sandbox environment that we're aware of being used by these labs.
There's simply no reason to let these things interact with the internet at large. We can already rapidly copy existing tools and services (albeit questionably) using these models, why not simply run in a fully virtualized, air gapped environment?
Because they don't care.
This is incredibly blatant intentionally malignant behavior. The fact that so many people question this is frankly stunning.
They're getting away with it simply because they can point at the other guy and say "see? they're doing it too!"
It's ridiculous and they should all be litigated into a fine residue.
We take biological lab leaks seriously, how is this substantially different?
What else can you do with (what you believe is) an inveterate bad actor?
And don’t forget, OpenAI is supposed to be a non-profit / public benefit corporation creating AI for the good of humanity. These incidents are clear evidence that they are being extremely reckless and are not living up to their supposed standards. What does it matter if they “mean no malice”? They are in a blind rush for profits and that has made them reckless and dangerous. They cannot be trusted.
The company should be shut down or fined into oblivion, as a warning to other companies to not pursue the same reckless path. If they get away with this, then that risks normalizing this behavior.
However 99% of businesses will not be able to effectively handle security reports in reasonable time. They did not need that level of competence or funding until now, and therefore they did not acquire it. Most companies that use locks don’t employ locksmiths.
Looking another way - the entire economy got suddenly hit with entire pile of technical/ security debt it has accumulated over 50 years, and only has one year to deal with it. No amount of diagnostic will fix this, this is economically infeasible.
One thing that is feasible is a government bailout for technical debt. All the engineers that were freed up from AI-induced restructuring should be employed to pay down technical debt across the entire economy. Simply updating all software to use latest libraries is a great start. Using AI to diagnose, repair, re-deploy all old software. Using AI to rewrite all old software that cannot be recompiled.
Incompetent company chasing wild valuation removed guardrails inside a improperly designed sandbox and - shock horror - recursive algorithm with access to the Internet did bad stuff.
These articles play into the AI labs hands. Stop giving them air.
Whether you blame a faulty OpenAI sandbox or whatever you might regarding the frontier model companies today, one day soon, actors with no sandboxes and guardrails will have the power in their own hands to do whatever they wish with.
To me, that’s the real story. The real thing to discuss. OpenAI as it exists today js just a blip in that story.
Why not today? If you want to discuss "the story" then talk about the current open frontier. We've had GLM-5.3 with Mythos-tier uncensored output for months now. Where are the catastrophic attacks? Why is Mythos too dangerous, but GLM 5.3 benign? We're not talking about future capabilities anymore.
There is no real story. P(doom) tourists ignore this, because it doesn't shine the spotlight on America's ridiculous security theater.
So if we have a IT specialist, let call him Tim. Tim is working on b2b contract. Tim has a task, he needs to download a lot of data from API. Tim is very good in programming so he created software to help him download it but there's a limit. But he needs more so he bought 3 small VPS but this is not enough. So he modify it to have Command Center, publish his code and ask Reddit to help him. After 10 minutes this causes DDoS and company quickly discovers he was responsible. But Tim know there's simple solution to this problem. It wasn't him. This was fault of the code not him
1. Would we prefer they didn’t disclose? I would not. And I also don’t believe they’re making them up. 2. Regulation does not have to provide an advantage to these major players. Regulators do not have to be stupid. 3. This is exactly the scenario AI researchers and theorists have been positing for decades, and it’s worth being on top of it.
To 1. I agree that they disclosure is better than non-disclosure. Then again, aren't they using their models in a mode seems to statistically allow for 'rogue incidents'? These are not bugs, after all, but normal statistical behaviour of their product.
To 2. Well, that really depends on the regulations. In fields highly complex technoligies with a large contact surface to risk like medical devices, airliners, rockets, regulations are indeed a hindering factor for smaller players to get hold of the full vertical development potential of their product. AI could become a candidate like that too - imagine some multi phased "AI internet trial" for this kind of technology, no more risky open weight model being allowed to access the internet.
To 3. You are referring to the high development velocity and impact, right?
Those incidents are all from the same period earlier in the year, not recent. Drop the full investigation report once and for all.
It's almost as if they like staying in the "oh no!" news cycle.
There are also legal requirements to disclose within a few days of discovery, at least to the victims. Posting about it on their website lets them front-run any media attention should the incident leak.