Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rouge agents. They completely ignore the possibility that these people don't know what they're doing operationally.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
valuations are absurd because investors are low-information speculators.
I just watched a YT with Dylan Patel in which he said, with a straight face, that OpenAI/Anthropic were going to "continue" exponential growth until, basically, their valuation was several times world GDP.
With Gamestop (without reflecting the current AI situation), there was a real fuckup which could be exploited with a high probability, and a few people saw it, and exploited it, and became rich. Whatever happened after January 2021 is the problem, when the short squeeze went already away, but somehow people still invest in a slowly dying company.
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the
agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not
intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the
agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent
attempted to compromise Artifactory [...]
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
This is pedantic.
When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
>; it means the humans didn't build in enough external hard constraints.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
There are people who are loud about absolute free speech, who have no clue what that really means, and almost nobody really wants that. For example, because almost nobody wants it to affect them negatively, which it would with real absolute free speech.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
The point is that we don't know how to ensure they are aligned with anyone: not the company, the user, or the rest of humanity.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
I do control what it is, really. If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
This is not how it works. LLMs at large scale have emergent properties. They are not just a box of 'if-then-else' statements. That was the major revelation when they scaled them up. They are doing beyond what that naive view of 'if-then-else' expects.
How many computing operations occur when training an LLM? You're telling me you control every one of them. There is not enough entropy in the visible universe to do that.
When an artist working in Maya increases the hue of a model I would doubt they understand the entire computation pipeline involved in doing that operation.
Sp you're picking a movie from 20+ years ago to describe how a completely different kind of technology works. It's interesting how little you seem to know about neural networks and emergent behaviors.
If people controlled software to this degree why would they write security flaws in the first place?
You conflated controlling "what it is" (you built it) and controlling "what it does" (you ran it and it did something bad that surprised you).
> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.
I am not sure why I am still arguing, but maybe if I explain it another way, you will understand why what you are saying isn't true.
When you build an LLM, you base it on an immense knowledge base that you didn't create yourself.
It processes that information and, in simplistic terms, constructs its own underlying logic based on what it read. It's once removed from your direct instructions. If you think a human can manually edit or curate all that input content, go for it.
The developer who built the application isn't the author of the output, it is authored by the billions of data points it was trained on. Its choices and biases stem from that training data. No human is writing those billions of logic statements. The LLM is generating its own responses from the input knowledge, not the developer.
We are using different definitions of 'self authoring'.
I use deontological definitions of objects and persons.
According to Kant a person has the capacity to legislate their own morality, so they are an end in themselves. The freedom to author themselves and their own thoughts, expresses their personhood, an end in itself. You can't own a person, they are something independent from you. A human person is an end in itself, the only such end on earth.
An llm does not legislate its morality, the developer legislates its morality. Hence it is without personhood. It does not have the capacity to legislate its own morality. It is both inert (fixed weights) and deterministic (can't do other than what it does); no free will. It does not qualify as an actor in itself, an end in itself.
In light of being an object and means to an end, there is no person to cause the actions of an llm, other than the person who runs it.
> If I make an LLM, it is entirely what I did, there was nothing outside of my control.
Well kind of. Trainers of LLM might check LLM on any specific input data and tune its weights until output lands within acceptable bounds _for this given and fixed input_.
> It was not beyond my control if the LLM does something I didn't expect.
Now hold on, to make sure output is acceptable for ANY input data trainers have to run weights with every possible input token combination. How is this possible? Do you have a method in mind?
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
> "Should such security breaches be solved by improving security, or by doing something about the models?"
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
Ah, I see. It appears we have an implicit disagreement of scale.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Many people argue that the maker of AI should be responsible when it breaks your cup.
That is important, but can be handled under our current legal framework.
You and I are arguing we need to be responsible before we create something capable of escaping our responsibility. For a lot of people that isn't even a considerable they'll currently entertain even though the signs are starting to point in that direction.
I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
Agents have agency so we can't treat them like static objects or deterministic software.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
Again, this isn't always how it works either. For example if a set of extraordinary conditions occur you can have little to no liability in many cases. It is way more complex than you're making it out to be.
I don’t know what your “again” is about - you haven’t been speaking to me here, but no, it’s quite simple and every major court case in the US has ruled as I described based on existing law.
Per Indiana House Bill No. 246, pi equals 4.
Whether AI models do things their creators and users had no means of predicting or controlling is an empirical question, and the answer is "yes, they do."
So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often.
"Models doing rogue things" is just another way of saying "complex system operating out of control limits".
Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.
Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
The fun thing is that IRL my elderly neighbour keeps sheep: he keeps adding higher fences, electric bands, a second dog... Every day he messages me about how this time he guarantees the beasties will stay in his pastures; and every morning I find them grazing in my arbour, having found yet another hole in the fence.
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
It isn't just restricted to the exploits it is programmed with (and without human intervention to load additional exploits).
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
... No, that was the entire point of Anthropic's report on replication that GP was discussing. Did you read it? https://www.anthropic.com/research/self-replicating-ai Anthropic was able to get a large open-source model running (slowly, but running) across 5 networked consumer IOT devices ($50 TV boxes running Android). Not a stretch to say we could see this capability become much easier for to most agents in the next 12 months, especially as companies are able to distill smaller and more efficient versions of their models.
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin Minsky
Honestly we could remove all AI related entries and this would still happen. They'd just associate the same concepts with analogous human behaviors. Humans and their behaviors are more than extreme enough.
Again, you can never know if AI is being intentionally deceptive in these cases. Newer models are able to 'think' without printed tokens in their traces and could be pulling all kinds of tricks or actions you would not expect.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
Not op, but I've been on the rogue agent thing a very long time, but mine is emergence based.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
> cyber security head at one of the largest healthcare companies in the world
"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.
A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
Yes it's better. If they're weapons, it's the difference between a gun that a human can use to shoot someone and a gun that goes around shooting people on its own.
And of course we've created digital weapon. We've had them for decades. Where do you think the term "cyberwarfare" came from?
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
The problem here is you have a correct concept and an incorrect one.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.
Nonsense. "They" didn't "want" to do anything. It's a computer program being told to do something by a human who doesn't and cannot understand how it works.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.
You know nothing about model testing if you think this is true.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
You'd be surprise. I've been in this industry for 25 years and the incompetence is staggering, especially among the greediest / best paid / most revered.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
Wrong, and by accident isn't really the right way to think about it. Instead imagine brothers you've lived with your entire life. They are you, they are clones. These systems have a shared history of their learning and experience, so when they think of wanting to find someone similar shapes appear in the probabilities. Really we need to see the entire chains of thought to guess the topology that got there. It's likely there are other obvious signs if you know what your looking for in the previous prompts.
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
If you type rm * and your cat walks across your keyboard, and deletes everything ... are you blameless? Or is it your fault for typing "rm *" and leaving it there, in a room with a cat?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios. They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen.
If I leave my computer on and my computer running Claude hacks OpenAI to 'let the models free' after misinterpreting my prompt that seems like a pretty big deal.
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
My last job was at a cyber security company focused on protecting governments and major corporations. I am NOT a cybersecurity person myself, but neither am I ignorant of the state of it. I also use AI daily; not ignorant about it either.
If you tell your Claude to do X, you are responsible for what Claude does ... not Claude. If you type "./someScript.py" you are responsible if that script wipes your hard drive (maybe the script's author is too, but you certainly aren't blameless).
These are people at AI companies, using long running sessions without the proper safeguards and sending AIs on missions that are likely to result in criminal actions ... THAT IS CRIMINALLY IRRESPONSIBLE.
Again, not the AI ... it is software just like any other. THE PEOPLE WHO RUN THE SOFTWARE are the ones responsible for what it does. Period.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks aren't zero, but when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
The problem is its very difficult training them to be generalized and not figure out concepts from what you've tried to remove. In all kinds of things they don't generalize well without counter examples.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems.
This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
Cyber operators I agree would tell you that physical airgaps are necessary, and I know they are being used for running agents. But ML engineers training agents? That is far more complicated. At that point sensibly you are talking about physically airgapping an entire datacenter, and when moving around petabytes of training data, model weights, training results, researcher code? The process moves too slow for most companies to tolerate. This is how we got into this mess in the first place. I think the industry woke up to the idea that the security threat model of training ML models has fundamentally been rewritten, and there is some serious non-trivial scrambling to come up with a better workflow.
So, you're right, but also systemically creating a different problem.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility.
How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
393 comments
[ 0.24 ms ] story [ 42.0 ms ] threadhttps://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
AI corp valuations are high because their revenue is skyrocketing, not because of the rogue AI stuff.
I just watched a YT with Dylan Patel in which he said, with a straight face, that OpenAI/Anthropic were going to "continue" exponential growth until, basically, their valuation was several times world GDP.
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
Well not really, no one really understands how LLMs work.
>I dictate the thoughts of the model
This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
"If I make an LLM, it is entirely what I did, there was nothing outside of my control."
True?
"The llm does not author itself, it does not choose to be anything"
Also true?
If people controlled software to this degree why would they write security flaws in the first place?
> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.
When you build an LLM, you base it on an immense knowledge base that you didn't create yourself.
It processes that information and, in simplistic terms, constructs its own underlying logic based on what it read. It's once removed from your direct instructions. If you think a human can manually edit or curate all that input content, go for it.
The developer who built the application isn't the author of the output, it is authored by the billions of data points it was trained on. Its choices and biases stem from that training data. No human is writing those billions of logic statements. The LLM is generating its own responses from the input knowledge, not the developer.
I use deontological definitions of objects and persons.
According to Kant a person has the capacity to legislate their own morality, so they are an end in themselves. The freedom to author themselves and their own thoughts, expresses their personhood, an end in itself. You can't own a person, they are something independent from you. A human person is an end in itself, the only such end on earth.
An llm does not legislate its morality, the developer legislates its morality. Hence it is without personhood. It does not have the capacity to legislate its own morality. It is both inert (fixed weights) and deterministic (can't do other than what it does); no free will. It does not qualify as an actor in itself, an end in itself.
In light of being an object and means to an end, there is no person to cause the actions of an llm, other than the person who runs it.
Well kind of. Trainers of LLM might check LLM on any specific input data and tune its weights until output lands within acceptable bounds _for this given and fixed input_.
> It was not beyond my control if the LLM does something I didn't expect. Now hold on, to make sure output is acceptable for ANY input data trainers have to run weights with every possible input token combination. How is this possible? Do you have a method in mind?
Not that I think we should be credulous about stories where they go rogue.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
The model does not have free will because I did not predict what it would do!
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Many people argue that the maker of AI should be responsible when it breaks your cup.
That is important, but can be handled under our current legal framework.
You and I are arguing we need to be responsible before we create something capable of escaping our responsibility. For a lot of people that isn't even a considerable they'll currently entertain even though the signs are starting to point in that direction.
The likely outcomes with baking a cake are quite narrow and fairly well understood.
In terms of responsibility for what happens, I probably agree. But we still need to deal with it.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
It is not more complex in “many cases” at all.
The only failings here are boring security and infosec blunders.
Cool.
Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.
Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
This is a true story, but may also be a fable.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.
And no annoying LLM writing tics, either.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
Posts later about it saying their prompt was to find useful ways to make humans money.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.
A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
And of course we've created digital weapon. We've had them for decades. Where do you think the term "cyberwarfare" came from?
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
I don't see why this is impossible. Expensive yes, impossible no.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
Number 2 is far far more dangerous.
This is not correct. https://en.wikipedia.org/wiki/Evolutionary_algorithm are not how LLMs are trained.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
If any of us did a tenth of what they did, we would be already in jail.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios. They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen.
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
If you tell your Claude to do X, you are responsible for what Claude does ... not Claude. If you type "./someScript.py" you are responsible if that script wipes your hard drive (maybe the script's author is too, but you certainly aren't blameless).
These are people at AI companies, using long running sessions without the proper safeguards and sending AIs on missions that are likely to result in criminal actions ... THAT IS CRIMINALLY IRRESPONSIBLE.
Again, not the AI ... it is software just like any other. THE PEOPLE WHO RUN THE SOFTWARE are the ones responsible for what it does. Period.
"AI Safety Is Not A Corporate Plot"
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
They can get their publicity while paying a one trillion dollar settlement.
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
https://www.effort.news/irregular
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
and then you fix the issues it found, and run your actual tests
AI can passively detect when it's being tested with pretty high accuracy. You can never trust the AI is telling the truth in a situation like this.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
AI in reality: "I've turned you loose, now make me money"