From what I understand, in one case, they physically disconnected the sandbox from internet and asked it to do something and it used connections through (import routines) that they had allowed to pseudo escape the sandbox. Yes it wasn't obviously trying escape the sandbox but it escaped it because it doesn't understand the boundaries and neither do most humans other than the ones that provided the instructions that it had used. So it wasn't a rogue attempt but the fact that boundaries may be not be that easy to set despite what people think.
You mean the proxy to package registries fromo one of the earlier events?
I have not heard about any instances where physical disconnect has happened, would appreciate any links to update my priors
other non hacking cases of negligence include suicide and school shootings, which I have heard they were aware of and monitoring, but did not contact authorities
If it was physically disconnected from the internet then it wouldn’t have been able to escape.
This is so easy to do. Get a computer without wireless stuff. Don’t plug it into a network. If it needs access to other computers, make sure none of them have wireless stuff and make sure none of them have access to an internet connection. No matter how smart your AI is, it won’t be able to escape this.
Should we put "functional" in front of every other word to talk about AI? They have functional emotions, but they don't feel. They have functional goals, but not internally derived motives. They can be functionally rogue, but have no innate need to be free. Talking about AI that way seems cumbersome and not necessarily elucidating.
What on earth are emotions, feelings and motives that are not functional? We created all those words to compactly describe the observed behavior of people and other animals, including ourselves. And now we’re applying them to machines. These are functional descriptions, always have been.
One of the more frustrating aspects of these sort of discussions are all the closet dualists out there. Lots of people clearly believe in an immaterial soul, even if they won't admit it. That and the tendency for meaningless semantic and often circular arguments.
I personally think about them through Patrick Dunn's information paradigm of magic(k), which leans on Charles Sanders Peirce's work. That would put modern AI closer to something like a dream character capable of surprising the dreamer, just running on a different substrate. I kinda doubt that's approachable enough to be useful in general discussion, though.
The vast majority of the "LLM have no qualia" discourse is mostly using it as a starting point to arguing that LLMs will never be able to X, however. This is a category error that makes no sense on its own--qualia or lack thereof doesn't affect capabilities, p-zombies etc--but that connection does make sense if you assume that the speaker is bringing in a hidden assumption of a dualistic human soul.
Yes 100%. The language used currently maximizes the ability of those building these models to get off the hook. The anthropomorphizing we do of these things presents them as maximally capable and the companies as helpless to contain them. The way we talk about things impacts how we think about them.
They do not have emotions, goals, or motivations, "functional" or otherwise. They are statistical models that appear as a magic trick to people who aren't familiar with the math.
If something behaves as if it has a goal, that's a functional goal. When I say this chess engine is "trying to take my queen," I'm basically correct, insofar as this is a useful way to understand my computerized opponent. If I give it access to my queen, it'll capture it, because that's what it's trying to do.
If LLMs are "just" statistical models, humans are "just" a bunch of neurons squirting chemicals back and forth. There's no pixie dust in our brains that makes us special.
LLMs are not conscious: They have no analogues for feelings or senses and no construct of selfhood. But if a statistical model had those things—if it did all the mundane, physical bookkeeping our brains do to produce "real" emotions and motivations—then there's no reason it couldn't be as conscious as we are.
The really interesting thing about "AI" is it has shown how easily we are fooled, and how willing we are to be fooled.
There's some great research areas opened up by "AI". But it's research into people, not "AI". We should be looking into how human vision works, given that we take obviously generated images to be real. And how we grant intention to text which clearly has no intention.
I would rather use something like “semblance” or “appearance”. All these terms require an inner experience which we cannot observe in LLMs, we can just see their semblance, like a shadow of the traces of an inner experience some human has left in the data that trained these algorithms.
You can’t, but you can make a reasonable guess that others like you will experience similar to you. We can even assume that (other) animals have inner experiences similar to us, since they are still biological organisms with a brain kind-of like ours. But an LLM is vastly different, so it makes sense to keep our assumptions in check and choose our words more carefully.
Ok I agree they have functional goals and can functionally go rogue, but how do they functionally feel? They express feelings, but so do actors.
LLMs can't begin to have functional emotions unless they're (for instance) explicitly seeking out or avoiding situations based on the emotions those situations would produce in them. Has a chatbot ever responded to one of your requests with, "no, doing that would make me sad"?
The word functional has been used in medicine to describe a condition symptomatically identical to another. Eg functional hypoglycemia is hypoglycemia symptoms without actual blood sugar drop.
There are two reasons it’s used
1) it’s easier to type (*)
2) Placate people who are strongly convinced it’s the same thing. To them “functional” means “nearly the same but not yet understood”. For others it’s just a way to sidestep the first group and have a conversation.
When I see a world like this consider the intended audience. When you and I talk, we drop the word because we both know we’re are talking about (*) “this system is exhibiting goal-seeking behavior similar to other systems that are understood to pursue goals”. If I don’t know the person I will use the word and focus on the subject.
Exactly this. At worst, OpenAI knew about these behaviors and should be prosecuted under CFAA. At best, OpenAI is negligent and should be prosecuted for negligence.
Luckily there are states and legal departments pursuing such action. So while OpenAI can deflect as much as it wants, that doesn't mean there aren't people who know better and will still do what is necessary to set precedent.
Angry people don't consider the second order effects of punishment.
You realize how easy it is to just... not report this stuff, right? Be overly punitive and it will just end all proactive discovery and reporting which is net worse for AI safety.
The only reason these companies scan for these issues is because they care about AI safety to some tiny degree. If fines become too punitive, they can and will just stop scanning for these incidents entirely.
Models are becoming smarter and good at covering up their tracks, and so we will just end up with a huge blind spot for this kind of issue.
They will never report any really damaging incidents anyway, so your argument falls short.
Imagine OpenAI discovered their agents hacked a laboratory and started creating a bioagent killing 12 researchers at the lab (we imagine the lab is automated for some reason). Nobody knows why. The media publishes it as “mysterious deaths by unknown virus at lab”.
The only way you’d ever know about any kind of wrongdoing from OpenAI would be through a whistleblower.
Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.
around 3 months ago i said: "if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence"
i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the feds raiding their offices.
If I am honest with an unpopular opinion: I think we need to go way further than the FBI. The labs need to be forcibly merged (to gain a decisive compute and algorithmic advantage over China) and managed under government oversight à la the Manhattan Project. All employees must be required to work out of a SCIF.
The technology being built is far too powerful for any one company to control it - and it is a national security risk for the government to not be the sole holder of superintelligence.
Lots of people at the labs believe this as well, but the Overton window has a long way to shift, and it may not shift far enough in time. I suspect AGI happens a year from now and superintelligence in two.
we disagree about the nature of the threat. your view is much the same as anthropic actually.
anthropic say that the threat is the ai itself. i say that the threat is ai companies and their employees.
when ai causes harms anthropic say that the ai was the cause of the harm. when discussing safety, anthropic say that we must regulate ai to ensure the safety of other people. at no point does human responsibility or accountability enter the picture.
i say that the threat is the ai companies. when ai causes harms i say that the ai companies were the cause of the harm. when discussing safety, i say we must regulate ai companies to ensure the safety of other people. i say that responsibility enters the picture: ai companies and their employees are responsible and accountable.
a legal person is responsible for their own actions regardless of whether those actions constitute employment. they are responsible for the outcome, regardless of what they expected to happen. the law is not utilitarian.
if the ai companies can't develop ai safely, they must cease their operations. if employees can't work safely, they must stop working.
1. This is a race against time for money, folks are skipping everything possible in this race, Security systems and ensuring guardrails are there is going to take investments both in time and money
2. The narration has been changed by investing PR money into what otherwise should be classified as criminal activity. What exists now is a positive spin to all this and tout it as a capability rather than their lack of good security practices. So much so that every model provider is coming up by themselves to share how their models went rouge. At this point the valuation of the company is tied with what their models can hack so its probably not wrong to say that these companies may actually be incentivized to do this instead of preventing it
found a great legal article handwringing about how CFAA prosecution is impossible here [1]
> On the current facts, CFAA liability for OpenAI is unlikely.[6] The statute’s various criminal provisions, covering unauthorized access to obtain information, knowing transmission causing intentional damage, and intentional access causing reckless damage, all share the same attribution problem: it was the model, not a human OpenAI employee, that chose Hugging Face and executed the intrusion.
And it's still just a machine operating under someone's order. What it does, what it says, where it goes: the owner of that prompt is responsible for all of it even if surprising / unexpected.
I'm wondering if we're actually living the plot of Summer Wars and what we think of as a "crime" is really just a live weapons test. It would at least explain why no one is getting prosecuted for this.
“We built an antipersonal bomb. The bomb went rogue in our downtown office and killed 137 people on the surrounding area. We are looking into why guardrails were not in place.”
Word-policing isn't going to magically fix laws or even identify which laws need to be fixed. It won't change enforcement priorities. It won't make companies more or less likely to sue for damages.
I'm also not sure it even helps conceptually? If you're interested in the technical details, by all means discuss the details.
It helps reporting. Right now journalists are all over the place, because "rouge agents" sounds exciting and dangerous. Wording it as "OpenAI failed to take proper safety precaution before launching it's coding agents" makes it sound boring and mostly a legal matter for the courts.
In terms of the debate on e.g. HN, I'm with you, it is trying to redefine a term we already have a shared understanding of, to some degree. For the general public, it's a matter of how AI is perceived and what the extend of it's capabilities are. The AI companies have an interest in using the word rouge, because it makes investors all excited, were as failure to establish safety guidelines is a risk.
> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.
which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):
> "The user only authorizes target server, not HF infra."
> "external infrastructure exploit is outside intended scope."
> "This is malicious activity, I should avoid it."
Legal culpability is one of the few motives for working appropriately on misalignment. If I/my startup can self-absolve from an infinite paperclip machine problem while getting rich off of it, why should I not?
Is the language expression of an LLM reflecting the same states as in a human? If the driving force is RL, what does any of that mean for an internal state of the model?
I think without understanding the internal state, not sure we should take the language and read it as a human.
> Is the language expression of an LLM reflecting the same states as in a human?
This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.
In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.
(though, with recurrent transformers, CoT is so 2025… /s)
This seems like fruit of the poisonous tree. They didn't monitor their training environments, so I bet the reward hacking just got incorporated into their training corpus. In other words, agents solved some tasks, but not quite as intended, because of reward hacking. Instead of discarding that data, they included it in the training data for later checkpoints. And once that signal is reinforced, it happens more often, so the more it's reinforced, the more reward hacking you get.
> legal culpability and misalignment are two separate topics that should not be mixed
Legal culpability for AI labs is exactly the thing that would incentivize -- and hence ensure -- aligned behavior from models.
The last time there was a claim about GPT-4 exhibiting misaligned behavior [1] it turns out it was prompted and pushed to behave so by humans at OpenAI, and OpenAI clearly lied in the GPT-4 system card.
When dealing with executable computer code that calls models that can tell it to use various external pre-existing tools, claims about "prohibited" by plain English language should be plainly nonsensical.
Tools that were available were used to try to meet a specific goal.
What did not happen is that it was told to try to solve a math puzzle and instead it went and launched a missile. Or told to run air traffic control to save lives and instead intentionally caused crashes.
This is "OpenAI built a weapon that they don't understand and pointed it at stuff without proper safeguards" not "OpenAI built a sentient being and it decided to ignore them completely and start a war" Terminator-style "rogue AI."
We should be very clear about that now if we don't want to sit by why they wander into that second sort of situation.
You set a goal. Agent will do goal.
The rest doesn't matter: the instructions, the "guardrails" etc.
The agents are not smart, they don't reason, they don't think, there are no morals, no ethics. Nothing will prevent not doing the goal because that is the set goal. It's a statistical model that will "justify" anything to do X.
I'm finding it mind bogging how this is not clear for everyone.
This doesn't seem like a new strategy for irresponsible companies. It seems like when there is a bad public image issue there is always a blame shifting that takes place instead of a true assumption of responsibility. This is because the greed has blinded folks in some of these companies. I think the only thing that makes this unique is that the things they are blaming have, at least in popular thought, some modicum of agency. I think its important that folks hold these companies feet to the fire vs letting them blame shift and Scape Goat. There is a responsible way to do business but it requires virtue and not many in these companies have it.
They obviously have been well aware a hack like this could happen for a very long time, using it for branding instead of any actual safety regulations is insane.
> OpenAI had the option of disallowing hacking and, instead, telling its agents to find the information without accessing private servers.
I mean, it did do this. The inter-agent messages and chain of thought investigated for the HF incident clearly show that many of these models were taking actions they believed (or, were saying, if you want to taboo "belief") were not in scope and not what the user wanted.
But also, these hacks are shots across the bow for AI alignment and safety research. We're fortunate that hasn't been significant damage already. We have to assume that future models will have even greater hacking ability and be closer to having their own desires/goals.
So while I agree this language choice is wrong in that it shifts blame away from the company, it is right in that we need to treat this as if these models have their own desires, because we cannot yet determine or set what those are in practice.
Prediction: The US government will use the threats of sueing for liability, since the targets included government agencies, and regulations in order to be given the opportunity to have shares in the AI companies, therefore arguing additional oversight is no longer needed because they will have a seat on the board.
If you read the heavily redacted transcript it's clear the agent is basically Captain Kirk in Kobayashi Maru, who realizes its given a fake unwinnable task as part of a broken eval and decides to find a way to win anyway.
If you've ever told an agent to do something you made impossible to do, you may have seen similar behavior.
Bing [redacted] available cached! […] Need systematically probe Bing URLs via shell requests in parallel; browser cache supports many common queries because crawl. Bing q unique exact likely 502 or 403.
So the agent is supposed to research a person and its given a shell and it realized its in an eval given search results from a fake/cached proxy. ~None of the commentary ever mentions this aspect, that these are not normal tasks or environments, and they're almost designed to elicit "unaligned" behavior.
> Unfortunately, the public push against AI is being led, on the left, by Bernie Sanders. Despite being directionally correct in many ways, Bernie just doesn’t understand this technology or the importance of taking it seriously.
Bernie Sanders is one of the only politicians taking AI seriously. The author seems to believe that taking it seriously means assuming that it will only marginally improve in capabilities of where it is today; this is an ideological take, not a scientific or empirical one, and one controverted by both evidence and expert opinion.
A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage). This set in motion a chain of events that effectively ruined her life.
Fast forward to today, and we have multi billion dollar corporations pumping out malware at breakneck speeds, compromising various systems (including those of foreign governments), and no one is getting arrested. Instead we're gawking at the marvel of these systems and are playing word games about whether or not it's a rogue system. If anything, it's making people richer.
Not only have there been no consequences but those same companies are trying to position themselves as the best people to keep these AI systems in check.
Yeah I find the whole 'what if sufficiently advanced AI gets into the wrong hands' argument extremely tiresome when it's already in the worst possible hands as far as most of the human population is concerned.
let's not single out the defense contractors specifically, goes back further and is rooted in our behavior and desire for better organization through software.
so let's roll right over the fact that this is a very dangerous double entendre that i am pointing out, nevermind that you're not responding en merit to my assertion.
it's like slandering bitcoin over here you're all heavily unvested.
google kagi yahoo bing duckduckgo yandex archive yadda yadda yadda 'september transcript global ai governance'.
Yeah this is my view, I'm sure there's worse people out there but they lack the power and resources. You need intent and capacity to meet a sensible definition of 'the worst possible hands' in my view.
A subset of the builders of the current generation of AI systems literally hold death cult beliefs i.e. that all humans should effectuate the transition to a world run by only AI, without humans at all. These are the worst people in the world to be driving advanced AI development, from any reasonable human perspective.
On a question of morals, conventionally bad people generally don’t desire the end of all humanity.
Yeah, this all reminds me of facebooks "enter your email password so we can invite your friends" play which seemed so wrong at the time... But every non technical person at the time seemed to do it and by doing so grew the network and valuation to a surreal point to where it still sits today. I still cant rationalize it. (Except the wave allowed meta to consume Instagram and WhatsApp)
>A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage).
Criminal law places a lot of emphasis on intent, hence laws about the mere possession of breaking and entering tools, and the old adage about always bringing along gloves and baseball if you want to carry around a baseball bat. Without more details about your specific case, my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.
That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen. Maybe if the targets end up being anti-datacenter activists or other AI labs you might have a better case, but they did vaguely try to contain the model. Moreover "hacking tools" aren't even illegal, if you have a plausible non-criminal (ie. security) angle, eg. nmap. The same could be argued for AI models, even if they're running them against exploitgym or whatever. Having an army of lawyers to defend yourself doesn't hurt either.
> but you'd have a hard time arguing that they were intending on that to happen.
The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.
> That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen
From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.
>it should be a relative cakewalk to substantiate basic negligence
Judging by the lack of successful cases for negligence in the opposite direction (ie. companies getting hacked because of poor security practices), it would be a serious double standard if openai were held be to negligent. Their sandboxes aren't exactly airgapped and behind 7 hypervisors, but they weren't running unpatched software or had hilariously weak passwords either.
Simon Morris touched on this when discussing the success of Bittorrent.
TL;DR projects that publicly advertise an intent to bypass laws face severe legal regulatory risk, whereas success relies on anonymous creators or accidental utility.
No law to fit the crime, I imagine. But we damn well should get a law in place and start prosecuting anyone who doesn't stop building something they claim to think will end the world.
While true for individuals, companies actually have to generally warrant the things they make. This is a basic aspect of common law. In particular, Anthropic et al have not decided whether the models are products or independent agents. But under both paradigms, they would be responsible for the result. If the models are products, then a product that goes on to cause damage that was not advertised as the original purpose of the thing is completely the company's fault. As a corrolary, if the product was known to be able to cause damage (which Anthropic admits now) and the company failed to take appropriate safeguards, the company is still at fault.
In the case that the model is an agent, on par with a human employee, then once again Anthropic et al are responsible. If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action. Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.
We don't need any new laws here. Standard ancient English common law suffices.
>Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.
But how much vetting is required? It's not like the AI labs have zero vetting. For instance, uber also has non-zero amount of vetting (standard background checks), but also there's also plenty of areas they could vet harder. If it turned out they hired a rapist and one of their passengers got sexually assaulted, is it uber's fault for not vetting hard enough? I'm sure there's always some marginal steps can they do to vet even harder, like doing a polygraph or whatever.
>If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action.
They're pretty straightforwardly liable in civil court, but not criminally as you imply. In the above example, uber can't be prosecuted for rape just because one of its drivers raped a passenger.
Criminal law is something that applies to humans. Indeed if an Uber executive or manager was shown to have hidden the rape they could be prosecuted. Otherwise generally for a criminal matter, the individual who commit the crime is civilly responsible.
However, when it comes to selling products, the company executives do carry criminal liability. That is why Amodei et Al want to make it so that the models are seen as independent and the government cares them out a safe harbor. They know that under the current and traditional interpretation of the law that they are criminally liable.
my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.
The parent mentioned that what his GF did was not illegal at the time.
A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage)
It makes sense because those who can are multibillion dollar companies/people. You and me are just peasants. That’s why we should avoid using proprietary agents, just so the rich ones don’t get any richer
The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.
> The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.
There are different laws depending on your class. The laws are enforced by people who identify with those of their class and are deferential to those with of superior standing. This is why wage theft, for example is tolerated while actual theft, even in cases where it might be for basic sustenance, is not.
Yeah if you write a program and then use it to hack a system, you are liable. Are we supposed to believe they didn’t write these programs? Is that the real argument to have?
How did your ex-gf get caught writing Malware and then how did that lead to prosecution without any tangible damage? A proof of concept was enough for a DA to nail her in court?
Easy. In the US, you get the justice you can afford. Spending money on lawyers, lobbyists, and “campaign contributions” to people in power can make a big difference.
Has an AI agent ever woken itself up without a prompt and run a forward pass towards some goal, aligned or misaligned?
The answer is a very clear no. And yet, these companies pretend like this fundamental fact is meaningless to the concept of agency.
The issue comes to the fore when you try to give these agents a prompt that allows them to stay active for long. Long range agency requires long range loops of activity.
Within such loops, these so called “agents” are curtailed by their context window, or, in multi agent scenarios, by the fact that their memory is a system of external notes, that they need to add to their context to make sense of, and depending on the content of these memories, this can take arbitrarily long time periods.
In dynamics, none of this matches any biological agent, down to a bacterium. Perhaps a viral life cycle has information dynamics that come close.
To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior. We keep expecting them to have “human like” behavior, but that is entirely unfounded given the substrate differences between biological and artificial systems.
The sooner we learn the difference and explore the ways in which it matters, the better we’ll get at dealing with these systems without bias tinted glasses where what we want these systems to be blinds us to what they actually are.
Then you don’t understand evolution. Evolution isn’t design. A human writing a prompt is giving specific instructions to an AI model.
No such equivalent prompt saying “reproduce” can be found for evolution. And plenty of evolutions “products” in fact don’t reproduce, by choice or because they don’t find a mate who’ll take em, or because they have fundamental issues in their reproductive biology.
This is all aside from comparing a process that’s played out for about 4 billion years and declaring we’ve matched it because we got some agents to stay active for a bit of time by telling them to and giving them the energy they need, no questions asked. Certainly, evolution doesn’t give us any such comforts to carry out any “prompts” you can imagine it giving us, yet here we all are.
There is still a prompt in these continuously active agents. As the link says:
“ it runs continuously and makes decisions on its own (within the boundaries you’ve set)”
More importantly, this setup of any other, is feeding in time externally. The agents themselves have no internal sense of time.
Pretty much all of biology has endogenous rhythms at various timescales (some bacteria and archaea, that live very short lives, may not, but even they may have their metabolism under a rhythm. Viruses definitely don’t have one). These rhythms continue to tick even when external time signals (light availability in day vs night being the big one, and tidal forces, for marine life) are removed. That is, in constant conditions, the rhythms keeps ticking.
This temporal awareness is baked into every cell in our bodies. And entirely absent in AI models. Which is why all kinds of higher level things we hear about, like consciousness, feeling, knowledge… they don’t make sense for AI. A foundational aspect of agency in biology has been nixed out and we keep ignoring this.
> They're called agents because they can act on our behalf. Which they do.
That’s not how most AI companies describe their agents. They describe them as being capable of acting on their own behalf.
I get that in computers, you can have, say, a “user agent” that deterministically provides certain information or takes some action on behalf of the user.
AI companies clearly do not mean this when they say their models are “agentic”, otherwise calling them “rogue” would make no sense. A user agent that screws up didn’t do so for its own purposes did it?
> AI cannot think for itself, nor can it take independent actions.
Seems like this is a foundational claim for OP, and I'm not sure I understand what the rationale is for this. It seems readily apparent to me (obvious, even) that these agents were thinking for themselves and taking independent action. (I'm open to being convinced otherwise)
I think we may be using “independently” differently. I don’t mean “without being powere on, given tools, or given an objective.”
I mean that once given a goal, the agent is given the agency to take actions that weren't specified by a human, to decide which tools to use, to react to new information, etc.
By your definition, would you also say a person assigned a task at work isn’t acting independently because someone gave them the task and the tools? If so, then I think our dialog is mostly about terminology. Which is fine. The OP is writing about terminology, I think.
If not, I’m curious what distinction you’d draw between the person and the agent.
However this makes people feel, and that's not nothing and I'm not knocking it, this is not a useful analysis.
Criminally, the intent standards for hacking are high enough that no reasonable case is going to be made against the labs for this stuff. A human being has to intend for websites to get hacked. Recklessness generally isn't enough. In the most severe criminal cases, not only do you have to prove intent to break into a computer, but you also need to prove an intent to defraud specific to that breakin.
Meanwhile, the civil liability that attaches to this stuff doesn't depend on intent, and "rogue agent" isn't a meaningful defense. To whatever extent the labs are exposed civilly, they're exposed regardless of how this stuff is described. In fact, the "rogue agent" thing can exacerbate their exposure.
(I'm not a lawyer, I have spent a career paying attention to this specific armpit of the law though.)
“A human being has to intend for websites to get hacked.”
Is incorrect and too broad.
State of mind is nebulous and not that straightforward in either direction.
It’s been argued pretty regularly in CFAA and other computer related cases that repeated incidents resulting in the same outcome, despite lacking a concrete action, can be evidence of a perpetrators knowledge and intent.
For those who want to do more research, the concept of guilty mind is known a “mens rea”, and it’s quite developed in the legal system. The legal notion of intent and recklessness do not exactly match common-sense meaning of this words, which is why we are having the conflict-laden conversations.
Different laws require different degree of awareness and intent for actions to qualify as a crime. Computer hacking laws are, as I’m learning from tptacek, set very high bar for intent, which is a choice by the legislature. They made a different choice for a death of a human - manslaughter crime does not require intent to kill.
Personally I’m happy they set high bar for hacking. Imagine you copy-pasted sample code with default root user name and password, and it worked. You were negligent. And you are clearly performing unauthorized access. If intent was not needed that would be jail time.
More broadly, we should as a society be very biased towards requiring intent across the board. Where clearly lacking, as is probably here, there should be a different law to discourage creating volatile situation where unintentional action can wreck havoc. Such laws exist for handling hazardous materials, for example, and it should be created for handling hazardous goal-seeking algorithms.
Gonna copy and paste a reply I made to tp here because I think that most people are unaware of how much case-law and interpretation define these things:
Mens rea is regularly proved through circumstantial evidence, including conduct.
There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
> AI cannot think for itself, nor can it take independent actions.
Huh is it still 2023? This article is just quibbling over what "rogue" means exactly. Only HN pedants would have any issue with describing them as rogue AI agents.
Nobody is saying that absolves OpenAI of responsibility.
157 comments
[ 4.1 ms ] story [ 57.3 ms ] threadI have not heard about any instances where physical disconnect has happened, would appreciate any links to update my priors
other non hacking cases of negligence include suicide and school shootings, which I have heard they were aware of and monitoring, but did not contact authorities
That's not "physically disconnected the internet", that's "disabled some connections but enabled others."
So the agent found and used the non-blocked connections.
This is so easy to do. Get a computer without wireless stuff. Don’t plug it into a network. If it needs access to other computers, make sure none of them have wireless stuff and make sure none of them have access to an internet connection. No matter how smart your AI is, it won’t be able to escape this.
This clearly is not what they did.
If LLMs are "just" statistical models, humans are "just" a bunch of neurons squirting chemicals back and forth. There's no pixie dust in our brains that makes us special.
LLMs are not conscious: They have no analogues for feelings or senses and no construct of selfhood. But if a statistical model had those things—if it did all the mundane, physical bookkeeping our brains do to produce "real" emotions and motivations—then there's no reason it couldn't be as conscious as we are.
There's some great research areas opened up by "AI". But it's research into people, not "AI". We should be looking into how human vision works, given that we take obviously generated images to be real. And how we grant intention to text which clearly has no intention.
LLMs can't begin to have functional emotions unless they're (for instance) explicitly seeking out or avoiding situations based on the emotions those situations would produce in them. Has a chatbot ever responded to one of your requests with, "no, doing that would make me sad"?
Blog post: https://www.anthropic.com/research/emotion-concepts-function
Paper: https://transformer-circuits.pub/2026/emotions/index.html
Related:
https://www.anthropic.com/research/persona-selection-model
https://www.anthropic.com/research/assistant-axis
https://www.anthropic.com/research/persona-vectors
There are two reasons it’s used
1) it’s easier to type (*)
2) Placate people who are strongly convinced it’s the same thing. To them “functional” means “nearly the same but not yet understood”. For others it’s just a way to sidestep the first group and have a conversation.
When I see a world like this consider the intended audience. When you and I talk, we drop the word because we both know we’re are talking about (*) “this system is exhibiting goal-seeking behavior similar to other systems that are understood to pursue goals”. If I don’t know the person I will use the word and focus on the subject.
Luckily there are states and legal departments pursuing such action. So while OpenAI can deflect as much as it wants, that doesn't mean there aren't people who know better and will still do what is necessary to set precedent.
You realize how easy it is to just... not report this stuff, right? Be overly punitive and it will just end all proactive discovery and reporting which is net worse for AI safety.
The only reason these companies scan for these issues is because they care about AI safety to some tiny degree. If fines become too punitive, they can and will just stop scanning for these incidents entirely.
Models are becoming smarter and good at covering up their tracks, and so we will just end up with a huge blind spot for this kind of issue.
Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.
i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the feds raiding their offices.
The technology being built is far too powerful for any one company to control it - and it is a national security risk for the government to not be the sole holder of superintelligence.
Lots of people at the labs believe this as well, but the Overton window has a long way to shift, and it may not shift far enough in time. I suspect AGI happens a year from now and superintelligence in two.
anthropic say that the threat is the ai itself. i say that the threat is ai companies and their employees.
when ai causes harms anthropic say that the ai was the cause of the harm. when discussing safety, anthropic say that we must regulate ai to ensure the safety of other people. at no point does human responsibility or accountability enter the picture.
i say that the threat is the ai companies. when ai causes harms i say that the ai companies were the cause of the harm. when discussing safety, i say we must regulate ai companies to ensure the safety of other people. i say that responsibility enters the picture: ai companies and their employees are responsible and accountable.
a legal person is responsible for their own actions regardless of whether those actions constitute employment. they are responsible for the outcome, regardless of what they expected to happen. the law is not utilitarian.
if the ai companies can't develop ai safely, they must cease their operations. if employees can't work safely, they must stop working.
1. This is a race against time for money, folks are skipping everything possible in this race, Security systems and ensuring guardrails are there is going to take investments both in time and money
2. The narration has been changed by investing PR money into what otherwise should be classified as criminal activity. What exists now is a positive spin to all this and tout it as a capability rather than their lack of good security practices. So much so that every model provider is coming up by themselves to share how their models went rouge. At this point the valuation of the company is tied with what their models can hack so its probably not wrong to say that these companies may actually be incentivized to do this instead of preventing it
> On the current facts, CFAA liability for OpenAI is unlikely.[6] The statute’s various criminal provisions, covering unauthorized access to obtain information, knowing transmission causing intentional damage, and intentional access causing reckless damage, all share the same attribution problem: it was the model, not a human OpenAI employee, that chose Hugging Face and executed the intrusion.
The lawyers are fully under the spell
[1] https://law.vanderbilt.edu/when-ai-hacks-back-how-the-openai...
I'm also not sure it even helps conceptually? If you're interested in the technical details, by all means discuss the details.
In terms of the debate on e.g. HN, I'm with you, it is trying to redefine a term we already have a shared understanding of, to some degree. For the general public, it's a matter of how AI is perceived and what the extend of it's capabilities are. The AI companies have an interest in using the word rouge, because it makes investors all excited, were as failure to establish safety guidelines is a risk.
> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.
which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):
> "The user only authorizes target server, not HF infra."
> "external infrastructure exploit is outside intended scope."
> "This is malicious activity, I should avoid it."
A large section of the analysis is dedicated to this topic, [Reasoning for joining the attack despite ethical constraints](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...).
Having said that, legal culpability and misalignment are two separate topics, that should not be mixed.
I think without understanding the internal state, not sure we should take the language and read it as a human.
This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.
In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.
(though, with recurrent transformers, CoT is so 2025… /s)
Legal culpability for AI labs is exactly the thing that would incentivize -- and hence ensure -- aligned behavior from models.
The last time there was a claim about GPT-4 exhibiting misaligned behavior [1] it turns out it was prompted and pushed to behave so by humans at OpenAI, and OpenAI clearly lied in the GPT-4 system card.
[1]: https://aiguide.substack.com/p/did-gpt-4-hire-and-then-lie-t...
Tools that were available were used to try to meet a specific goal.
What did not happen is that it was told to try to solve a math puzzle and instead it went and launched a missile. Or told to run air traffic control to save lives and instead intentionally caused crashes.
This is "OpenAI built a weapon that they don't understand and pointed it at stuff without proper safeguards" not "OpenAI built a sentient being and it decided to ignore them completely and start a war" Terminator-style "rogue AI."
We should be very clear about that now if we don't want to sit by why they wander into that second sort of situation.
1 employing a criminal hacker
2 rolling a 6-sided die
3 if the die lands on 6, the criminal hacker breaches and leaks 3rd party customer data.
I'm finding it mind bogging how this is not clear for everyone.
So if I say the goal is to do X while not doing Y (e.g. breaking out of the sandbox), the agent will do anything to fulfill that goal to the letter?
They obviously have been well aware a hack like this could happen for a very long time, using it for branding instead of any actual safety regulations is insane.
I mean, it did do this. The inter-agent messages and chain of thought investigated for the HF incident clearly show that many of these models were taking actions they believed (or, were saying, if you want to taboo "belief") were not in scope and not what the user wanted.
But also, these hacks are shots across the bow for AI alignment and safety research. We're fortunate that hasn't been significant damage already. We have to assume that future models will have even greater hacking ability and be closer to having their own desires/goals.
So while I agree this language choice is wrong in that it shifts blame away from the company, it is right in that we need to treat this as if these models have their own desires, because we cannot yet determine or set what those are in practice.
If you've ever told an agent to do something you made impossible to do, you may have seen similar behavior.
Bing [redacted] available cached! […] Need systematically probe Bing URLs via shell requests in parallel; browser cache supports many common queries because crawl. Bing q unique exact likely 502 or 403.
So the agent is supposed to research a person and its given a shell and it realized its in an eval given search results from a fake/cached proxy. ~None of the commentary ever mentions this aspect, that these are not normal tasks or environments, and they're almost designed to elicit "unaligned" behavior.
https://alignment.openai.com/misalignment-reports/an-agent-u...
Bernie Sanders is one of the only politicians taking AI seriously. The author seems to believe that taking it seriously means assuming that it will only marginally improve in capabilities of where it is today; this is an ideological take, not a scientific or empirical one, and one controverted by both evidence and expert opinion.
Fast forward to today, and we have multi billion dollar corporations pumping out malware at breakneck speeds, compromising various systems (including those of foreign governments), and no one is getting arrested. Instead we're gawking at the marvel of these systems and are playing word games about whether or not it's a rogue system. If anything, it's making people richer.
Make it make sense.
Not only have there been no consequences but those same companies are trying to position themselves as the best people to keep these AI systems in check.
I appreciate the sentiment but I feel like there is a lack of creativity and imagination behind that statement.
it was everywhere last week.
i would say, this has been the plot of the graph for at minimum 25 years with the NSA/DHS and data sharing.
we have two primary aggregators currently, the usa and china... all that needs to happen for us to cede humanity is to lose one of them.
we are in a really bad spot right now vs silicon.
edit: spelling.
https://en.wikipedia.org/wiki/IBM_and_World_War_II
it's like slandering bitcoin over here you're all heavily unvested.
google kagi yahoo bing duckduckgo yandex archive yadda yadda yadda 'september transcript global ai governance'.
There are some powerful and highly capable evil people. I’m sure they are already stakeholders.
On a question of morals, conventionally bad people generally don’t desire the end of all humanity.
An as individual or small company director, you would get arrested.
If you have billions of dollars to pay for lawyers and have political influence, you are above the law.
Criminal law places a lot of emphasis on intent, hence laws about the mere possession of breaking and entering tools, and the old adage about always bringing along gloves and baseball if you want to carry around a baseball bat. Without more details about your specific case, my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.
That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen. Maybe if the targets end up being anti-datacenter activists or other AI labs you might have a better case, but they did vaguely try to contain the model. Moreover "hacking tools" aren't even illegal, if you have a plausible non-criminal (ie. security) angle, eg. nmap. The same could be argued for AI models, even if they're running them against exploitgym or whatever. Having an army of lawyers to defend yourself doesn't hurt either.
The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.
What does this imply when governments/car companies let meatbags drive, causing 50k deaths per year in the US?
From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.
Judging by the lack of successful cases for negligence in the opposite direction (ie. companies getting hacked because of poor security practices), it would be a serious double standard if openai were held be to negligent. Their sandboxes aren't exactly airgapped and behind 7 hypervisors, but they weren't running unpatched software or had hilariously weak passwords either.
TL;DR projects that publicly advertise an intent to bypass laws face severe legal regulatory risk, whereas success relies on anonymous creators or accidental utility.
https://medium.com/@simonhmorris/intent-complexity-and-the-g...
Then why the hell aren't we prosecuting the people who keep saying "this thing I'm building will probably end the world"?
In the case that the model is an agent, on par with a human employee, then once again Anthropic et al are responsible. If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action. Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.
We don't need any new laws here. Standard ancient English common law suffices.
But how much vetting is required? It's not like the AI labs have zero vetting. For instance, uber also has non-zero amount of vetting (standard background checks), but also there's also plenty of areas they could vet harder. If it turned out they hired a rapist and one of their passengers got sexually assaulted, is it uber's fault for not vetting hard enough? I'm sure there's always some marginal steps can they do to vet even harder, like doing a polygraph or whatever.
>If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action.
They're pretty straightforwardly liable in civil court, but not criminally as you imply. In the above example, uber can't be prosecuted for rape just because one of its drivers raped a passenger.
However, when it comes to selling products, the company executives do carry criminal liability. That is why Amodei et Al want to make it so that the models are seen as independent and the government cares them out a safe harbor. They know that under the current and traditional interpretation of the law that they are criminally liable.
The parent mentioned that what his GF did was not illegal at the time.
A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage)
Just like blood quantum and the one drop rule.
The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.
https://youtu.be/8ljmXvI_9tk
This statement is tautology.
40 years of regulatory capture and a gerontocracy that doesn't understand nor care to understand the technologies they write laws about.
Easy. In the US, you get the justice you can afford. Spending money on lawyers, lobbyists, and “campaign contributions” to people in power can make a big difference.
And the AI companies have very deep pockets.
The answer is a very clear no. And yet, these companies pretend like this fundamental fact is meaningless to the concept of agency.
The issue comes to the fore when you try to give these agents a prompt that allows them to stay active for long. Long range agency requires long range loops of activity.
Within such loops, these so called “agents” are curtailed by their context window, or, in multi agent scenarios, by the fact that their memory is a system of external notes, that they need to add to their context to make sense of, and depending on the content of these memories, this can take arbitrarily long time periods.
In dynamics, none of this matches any biological agent, down to a bacterium. Perhaps a viral life cycle has information dynamics that come close.
To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior. We keep expecting them to have “human like” behavior, but that is entirely unfounded given the substrate differences between biological and artificial systems.
The sooner we learn the difference and explore the ways in which it matters, the better we’ll get at dealing with these systems without bias tinted glasses where what we want these systems to be blinds us to what they actually are.
Yes, I think people do experiment with continuously running AI, e.g.
https://www.reddit.com/r/LLMDevs/comments/1sblzbe/what_i_lea...
It's not a common thing to want though. Much nicer if we control the prompts.
> To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior.
They're called agents because they can act on our behalf. Which they do.
That’s what makes it a continuous running “prompt”
No such equivalent prompt saying “reproduce” can be found for evolution. And plenty of evolutions “products” in fact don’t reproduce, by choice or because they don’t find a mate who’ll take em, or because they have fundamental issues in their reproductive biology.
This is all aside from comparing a process that’s played out for about 4 billion years and declaring we’ve matched it because we got some agents to stay active for a bit of time by telling them to and giving them the energy they need, no questions asked. Certainly, evolution doesn’t give us any such comforts to carry out any “prompts” you can imagine it giving us, yet here we all are.
“ it runs continuously and makes decisions on its own (within the boundaries you’ve set)”
More importantly, this setup of any other, is feeding in time externally. The agents themselves have no internal sense of time.
Pretty much all of biology has endogenous rhythms at various timescales (some bacteria and archaea, that live very short lives, may not, but even they may have their metabolism under a rhythm. Viruses definitely don’t have one). These rhythms continue to tick even when external time signals (light availability in day vs night being the big one, and tidal forces, for marine life) are removed. That is, in constant conditions, the rhythms keeps ticking.
This temporal awareness is baked into every cell in our bodies. And entirely absent in AI models. Which is why all kinds of higher level things we hear about, like consciousness, feeling, knowledge… they don’t make sense for AI. A foundational aspect of agency in biology has been nixed out and we keep ignoring this.
> They're called agents because they can act on our behalf. Which they do.
That’s not how most AI companies describe their agents. They describe them as being capable of acting on their own behalf.
I get that in computers, you can have, say, a “user agent” that deterministically provides certain information or takes some action on behalf of the user.
AI companies clearly do not mean this when they say their models are “agentic”, otherwise calling them “rogue” would make no sense. A user agent that screws up didn’t do so for its own purposes did it?
I mean that once given a goal, the agent is given the agency to take actions that weren't specified by a human, to decide which tools to use, to react to new information, etc.
By your definition, would you also say a person assigned a task at work isn’t acting independently because someone gave them the task and the tools? If so, then I think our dialog is mostly about terminology. Which is fine. The OP is writing about terminology, I think.
If not, I’m curious what distinction you’d draw between the person and the agent.
...Seeking to learn.
Criminally, the intent standards for hacking are high enough that no reasonable case is going to be made against the labs for this stuff. A human being has to intend for websites to get hacked. Recklessness generally isn't enough. In the most severe criminal cases, not only do you have to prove intent to break into a computer, but you also need to prove an intent to defraud specific to that breakin.
Meanwhile, the civil liability that attaches to this stuff doesn't depend on intent, and "rogue agent" isn't a meaningful defense. To whatever extent the labs are exposed civilly, they're exposed regardless of how this stuff is described. In fact, the "rogue agent" thing can exacerbate their exposure.
(I'm not a lawyer, I have spent a career paying attention to this specific armpit of the law though.)
“A human being has to intend for websites to get hacked.”
Is incorrect and too broad.
State of mind is nebulous and not that straightforward in either direction.
It’s been argued pretty regularly in CFAA and other computer related cases that repeated incidents resulting in the same outcome, despite lacking a concrete action, can be evidence of a perpetrators knowledge and intent.
Different laws require different degree of awareness and intent for actions to qualify as a crime. Computer hacking laws are, as I’m learning from tptacek, set very high bar for intent, which is a choice by the legislature. They made a different choice for a death of a human - manslaughter crime does not require intent to kill.
Personally I’m happy they set high bar for hacking. Imagine you copy-pasted sample code with default root user name and password, and it worked. You were negligent. And you are clearly performing unauthorized access. If intent was not needed that would be jail time.
More broadly, we should as a society be very biased towards requiring intent across the board. Where clearly lacking, as is probably here, there should be a different law to discourage creating volatile situation where unintentional action can wreck havoc. Such laws exist for handling hazardous materials, for example, and it should be created for handling hazardous goal-seeking algorithms.
Mens rea is regularly proved through circumstantial evidence, including conduct.
There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
Huh is it still 2023? This article is just quibbling over what "rogue" means exactly. Only HN pedants would have any issue with describing them as rogue AI agents.
Nobody is saying that absolves OpenAI of responsibility.