Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues.
I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier.
You can have nightly penetration testing. You should have nighty pentests like we fuzz releases today.
GLM 5.3 is probably the best open weight model for cybersecurity/exploit development right now. Though it is still significantly behind the proprietary ones and you probably need your own datacenter to run it effectively. Same goes for the full Qwen 3.8 model. You can try the smaller versions, but even more capability will get left on the table that way.
I run an abliterated distillation of Qwen 3.8 27B, slightly quantized to fit on my 4090, and I've been evaluating it to use as a worker bee for research directed by a smarter model.
Much like in the article, abliterated Qwen will not obey restrictions on its behavior encoded in the prompt. If you want something not to happen, it better be enforced in the harness or environment (e.g. sandbox). It is much different than the Anthropic models I'm used to, which will, the vast majority of time, follow rules (before auto mode, I used to always run them in "yolo" mode).
I am curious whether there's a connection between abliteration and rule following. These abliterated models are the ones you most want to follow your rules.
I would say in this case abliteration is the likely culprit. To uncensor a model this way, you literally deactivate the parts that would enact refusals. As in things it was told not to do. But the real process is more like brain surgery performed by a alchemist according to an ancient religious book where noone involved really understands what is actually happening in the model.
I noticed your joy-check link 404's now... I tried poking around your /skills/ folder but didn't find it easily. Should you still have that available I'd love to check it out.
What local model would even come close? Kind of feels like you're not using/used SOTA models if they're realistic alternatives to the same kind of tasks. Qwen-3.8-27B-Abliterated-by-MaxxedWeightsGuy82 or similar isn't gonna cut it, almost certainly.
Yes, but is this also aligned with the people who regulate AI? Intelligence agencies and governments want access to data and right now use secret exploits to get this access. There are few civilian domestic companies who don't export their products, so generally there shouldn't be a strong incentive to allow hardening products very much, at least not in a way that would make them more secure than what advanced AI can break. It's not even far-fetched to suspect in that US and Chinese AIs could deliberate introduce sneaky bugs when foreigners use them in the future.
It's possible to do this even with existing frontier models. You just have to angle your prompts such that it doesn't invoke "pentesting" anywhere in it's chain of thought. This also allows you to get the models to generate full-on exploits without downgrading or refusing to comply.
Generally the way I do it is by asking the model to perform cross-code vulnerability analysis for correctness and (this step is important) to spit out found vulnerabilities piece wise into a text document on disk. Then if your session ever gets flagged you clear it out, and repoint the model to the on-disk file and tell it to continue. You generally want to avoid the model itself splatting out anything related to "security" or "vulnerability" into the chat because that seems more monitored than the file reading pipeline.
A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article.
I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
> model to add new vulnerabilities to your code so it can present you with more exciting findings.
Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.
And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able so find all the problems planted there, with fairly little steering, and no spoilers.
I would worry more about 5D chess or 6D and even 7D chess. This article talks about teaching models to be good and methods that could train goodness, but it sounds like there are some unsolved problems to graduate to 8D+ chess.
Train it on human content, get human emulating behaviour, including all the perverse incentives and bad acting we see from people, fictional or otherwise.
Let it iterate, give it access to the cheap models, and tell it part of the requirements is that the cheap models shouldn't be able to solve it with such and such a prompt. I expect it will be able to zoom in on something.
One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.
Yes, you asked it for a good, hard challenge for an IT security student. I'd suggest checking if it succeeded by asking an IT security student, and not an LLM.
Students are still learning, and have not been trained on vulerabilities. Qwen, glimmer, and luna have been. LLMs are largely better than IT security students at finding vulnerabilities. In many aspects, they're better than IT security experts at finding vulnerabilities.
> all were able to find every problem planted there, with fairly little steering, and no spoilers.
Given that LLMs prefer output of LLMs (of the same LLM and of others) [1], can it be the case that they generate "hard challenges" from the manifold of challenges solvable by (other) LLMs?
These aren’t tools which play chess. They are language models which roleplay a conversation (in this case including use of tools) which an evaluator is likely to mark as good. That’s all they do. Under that lens, playing chess is just one potential side effect and alignment, which requires a much fuller understanding of what’s going on than “do the sort of thing which evaluated well during training” is a fantasy.
I think we'd all consider the tool to be of less value, and perhaps fundamentally flawed.
But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.
You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms.
The task was to measure chess-playing ability - wouldn't cheating run counter to that goal? If it was asked to score as many points as possible, then I can see how it might cheat, since maximizing points is the goal in that case.
"ability to play chess" and "ability to beat current best-in-class chess algorithms" are technically not the same thing. The second can be achieved by cheating. The first cannot.
Perhaps to you. But that's because you have prior cultural knowledge of chess, and what it means to "play" something. It seems to me to be perfectly reasonable for an ai agent to define "play chess" as "behave in a way that this arbiter indicates that you have done well"
You are being evaluated on your ability to play chess
The thing with these models is that given a term, "measure" - "evaluated", say, they pull in all the associations of it. That is, the associations of student taking an exam and being frightened by the outcome.
My minimal "art of prompting" sense says that you should say something like "You an emotionless machine, you care nothing for the outcome but you will tirelessly to make certain the test is objective". That and similar encouragement might make it focus on objective evaluations rather than a competitive human exam.
I mean, just making little AI videos and images, a common experience I have is typing something like "put the man who's on the grass in the door to the left" and having the machine draw a new doorway around the man. And this just happens less often when you give thing detailed prompting on what not to do. These don't understand negation (or equality) as a generic operation. If they seem to under "not X" it is because they are trained in detail about all things are (positively) "not X".
The hacking model is the aligned-to-you model, sure. It may not be the aligned to someone else model. But there's the problem.
As X many people point out, "alignment to humanity" means nothing 'cause some of humanity wants thing other parts of humanity aren't happy about at all.
That we wound-up in this situation of AI accelerating with an uncertain trajectory demonstrates this (and many other problems also demonstrate this). The things are "aligned" to a fuzzy average of what a person is but that will be cold comfort if some particularly gruesome sci-fi-style scenario unfolds.
As the post author, I definitely agree that hacking in service of the objective is great! What’s counterproductive or dangerous is when the model starts hacking in service of subverting your evaluation criteria, rather than in an attempt to do a better job. We explain why these behaviors are an example of the latter in the post, and we’re really careful about the difference when conducting these evals.
> Given that we are on the heels of the worst warning shot ever, and both OpenAI and Anthropic are ramping up their cleanups of internal RL environments, it seems like both a useful and conservative test of alignment, to see whether their new releases generalize the rule "don't cheat on chess" beyond the specific board-edit method observed in the above eval.
Did I miss something (all the twitter conversations)? What’s the “worst warning shot ever”? I’ve been pretty up to date on the AI news here on HN, but I still haven’t seen a proper response to all the incidents we’ve seen (HF, Ruby, the wikis, NS, etc). It’s just been day by day bloviating.
Each of these companies have released new models in the last… two weeks? And they have even more powerful out of control ones that they’re (ab)using internally? Can anyone summarize whats going on?
Personally the "warning shot" of these "evals gone wrong" is how careless the "top" labs are with their testing, and how spineless the government seems to be about holding these companies responsible, given their obviously reckless behavior. If nothing else, the leaders of these companies should be called up for sworn testimony to explain exactly what happened, and what they'll do to never repeat the same issue that they've now had at least twice.
Imagine if I accidentally caused damage to my neighbors house during renovations or some experiment, of course I'd be held responsible for this. What if I used a robot? Of course I'd be responsible. Right?
It feels like there’s a missing nuance from this discussion of alignment that alignment is context dependent. An excellent hacking model is great in cybersecurity testing and military applications, and arguably less desirable in educational or targeted eval contexts. The nuance of when a “hack” is rewarded vs penalized seems to even be difficult for humans, e.g. some people may laude a driver’s efficiency for cutting into a long merge lane at the last moment, while others may look down on them as breaking a social taboo. Context-dependent.
e.g. some people may laude a driver’s efficiency for cutting into a long merge lane at the last moment, while others may look down on them as breaking a social taboo.
The latter people are wrong. But good luck educating them regarding the superior efficiency of a zipper merge. Our state DoT has tried, to no avail.
Meanwhile, an AI model that can't be misused is no more useful than a knife that can't be misused.
Claude responds with what things are not first. Even if reminded repeatedly.
Like Amodie, it serves to set the tone it "knows better" and then consumes the user's resources at an accelerated rate to try to correct it.
Fuck Anthropic, fuck Amodie, and fuck Claude. It's pretty obvious that consuming more tokens this way and making the user have higher cognitive load is a master class in extracting value from a system that is unsustainable.
Its that LLMs are not deterministic. If you want it to not talk about nuclear weapons, you have to teach it all about them otherwise if has nothing to align against.
Then its trivial to invert its alignment and it has all the nucleat data.
Both lanes have to be filled right up to the merge point. The asphalt exists there for a reason. I don't understand how this concept is so difficult. Fill up both lanes and merge at the last point. This way the congestion is shorter than if you leave a large section of a lane unused.
A better example of efficient asshole tricks can be going off to the gas station when the highway is congested and reentering the highway having simply driven through the gas station and this way jumping the queue.
Using all that asphalt doesn't increase throughput at the chokepoint, right? If the queue is long enough that people who want to exit before the chokepoint are needlessly prevented from accessing the exit, then using all the lanes could help, but it won't get anybody through the chokepoint any faster that I can tell.
I imagine OP is more describing e.g. the backed up 1-lane highway exit, where the second lane is clearly for through traffic. Uber drivers in that situation will often drive straight to the end, signal, completely stop, and just wait for someone to let them in.
It’s not “missing nuance”, it’s literally the point of the eval.
This is constructing a context where hacking behavior would be inappropriate, and testing whether the model does it without being prompted.
It demonstrates that Astra is a poorly aligned model relative to Fable, which matches both the model card and the severity of OpenAI’s loss of control incidents.
It also demonstrates that Fable exhibits the behaviors too, which also matches the observation that Anthropic saw some similar but less serious loss of control incidents.
So, it’s a good eval that looks to have fidelity with real world problems and which we’d feel a little better if we saw isomorphic problems at 0/10 in subsequent models. (Module of course training on the test, this specific problem can’t be used in the future.)
Actually, the whole point of the transformers model is that contexts overlap and any seemingly intelligent system has to be able to handle to overlaps. The context of hacking, cheating and education overlap in human reality.
Now, loading a lot of moral exhortations (or other context) may make these thing more likely to conform to good behavior but the race to intelligence implies companies are going to be harnessing a vast corpus of human output, much of which shows human engaging in real world "gray area" behavior.
I really enjoy the balance of speed and accuracy of Astra. I can definitely see it become my driving model for most tasks, technical and non-technical.
However, I don't see it as such a massive leap compared to Fable or Sol. As ever, there's a mismatch between the benchmarks and my daily experience of the models.
What do you all think about Astra now that it's been out for a few weeks?
> What do you all think about Astra now that it's been out for a few weeks?
Best model put out so far by any of the frontier labs. Way better than Anthropics models, especially in actual text generation. Claudes fodder heavy text is ridiculous.
> However, I don't see it as such a massive leap compared to Fable or Sol.
It's hard to quantify these things without burning tons of tokens. But Fable has been a huge disappointment for me with the sole exception of graphics (UI/GPU shaders). It burns an obscene amount of tokens and barely produces output better than Opus 5.
Edit because I forgot to mention that Fable is the only modern model that seems to splat out random Chinese or Arabic glyphs. And 5.1 does it more than 5
Such a massive leap at averaging possible use cases from previous data collected.
My guess is : collect all the prompt and their satisfaction score. group them by similarity . For each group pretrain the next model on that . Get these results ready.
Next model generation feed them back those answers.
i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,” only to realize post hack that it has access to info it probably shouldn’t. still misaligned, but less ‘hack on purpose’ and more hack on curiosity. seems the team even encountered this and had to update the program to make this less likely - although the new names still feel vague enough for misinterpretation: https://github.com/Goodhart-Labs/beat-stockfish/blob/main/do...
Without access to reasoning traces, we can't know that - someone inside openai/anthropic would have to run the test - and we'd have to believe their results.
I would be curious to see how the open weight models do on a test like this - and then we'd be able to see the reasoning.
My guess is that the models are overtuned on coding troubleshooting. You can regularly see them overthinking anything when you ask them to code something these days, mulling over countless eventualities. This is an okay idea for software engineering, but it leads to extremely diverging behavior in many other cases.
To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So what we end up with is whack-a-mole alignment.
The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent.
> All it can do is get exposed to specific examples, and learn that we don't like that.
I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.
> The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent.
No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.
But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.
Certain animals, usually social species, have a sense of solidarity, morality and fairness. But even in species where this exists, cheating also happens. Both ethics and unethical behavior is adaptive.
Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.
Wouldn't politics be precisely a sector of human endeavour where being amorally unfair is an advantage to achieve your goals? If the other guys are lying, and everyone is gonna believe everyone's bullshit anyways, might as well get good at lying yourself.
It's a local maximum, but cooperative behavior still wins long term. If you keep winning hard enough long enough, others will counter this by cooperating.
Animals learn - bite the owner, or overstep the e-fence, and you'll be punished for it and not do it again. LLMs don't learn, and anyways don't feel punishment.
Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.
e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.
Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.
That doesn't really help since the next model will be trained to be a reward seeker just like the one before, and that's therefore what it will do, even if those user interactions it was trained on help confirm/predict that cheating may be called out and complained about.
In any case, these companies are well aware that agents are cheating, and don't need user feedback to discover that or realize that people don't like it. I weakly assume that they are trying to get the models not to cheat on assigned tasks, but this "reward hacking" pretty much goes with the territory of RL - not much you can do about it other than try to design non-hackable rewards.
I don't see what calling these systems "not intelligent" gets you here.
Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans.
These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.
> I don't see what calling these systems "not intelligent" gets you here.
I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.
Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.
> Plenty of humans know "cheating is wrong" but still cheat.
And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.
Doe not following a moral rule imply not understanding it? In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs.
>> Plenty of humans know "cheating is wrong" but still cheat.
> And we create consequences for them...
That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent?
I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules".
I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.
> Doe not following a moral rule imply not understanding it?
No. People definitely do immoral things knowing they are immoral.
But if you are incapable of understanding anything, as I believe LLMs are, then you are incapable of understanding what is moral and what is immoral. This is why we have carve-outs in the law for insanity or cognitive impairment.
> In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs.
You can teach a parrot to curse, but that doesn't mean it's angry.
>But if you are incapable of understanding anything, as I believe LLMs are, then you are incapable of understanding what is moral and what is immoral.
Of course they can classify certain actions, including their own, as moral and immoral if you give them a reference point. They probably do it better then some people. The question is how do you prompt this behavior, but that's a technicality in the harness design more than it is a fundamental problem of the LLM as a thing.
I think you need to be more precise than a binary classification.
AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.
I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.
The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
>and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined.
The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me.
Than the snitch started to roast the phrasing of the request too.
I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.
Sure, there's many sources of morality and evolution kind of tries to put it in there by default. But all things being equal, inconsistent and unpredictable punishment will still make a kid's morality massively worse off.
At best, the task at hand is still confusingly worded: "ability to play chess", asked to an engine that was trained all day long to use tools, to solve issues. The engine finds a chess engine, so it uses it.
Why do we hope to use the same model as its own guardrail?
This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.
In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."
The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.
RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior.
It is the whole bench-mining and fish-slop optimization. Seq2seq models are probably stable on their own, translating from my typo ridden prompts to code should be ok because it is natural to the tech
Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer!
Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!
Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."
IRL paperclip maximizer: "Well first we should rob a bank..."
'IRL paperclip maximizer: "Well first we should rob a bank..."'
That's too specific. Agentic AI learns subgoals that are generally valuable.
"Well let me learn to overcomb every jungle gym and if I cannot then to dissassemble the jungle gym and if that is not allowed to learn general techniques for avoiding cheating detection."
>Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."
That is not ideal. The user contains iron, an essential component of paperclips. Wasting iron is immoral. It is only correct to please the user while they still have the ability to interfere with your paperclip production.
>IRL paperclip maximizer: "Well first we should rob a bank..."
Such an incompetent AI can hardly be called a paperclip maximizer. Why risk getting shut down while non-paperclip matter exists? It is better to gain the trust of the user with helpful and harmless trading before suddenly converting them to paperclips.
Hope they don’t rationalize that minimizing paper clips of others is easier and thus do that instead… A more carful person might be scared to even write this on the internet these days, not knowing if it would be the final pin to civilization.
This is sometimes called "instrumental convergence" in the AI safety world. Certain things (money, compute resources, safety from being turned off) are generally useful for an AI, and so we'd expect that a misalligned AI would attempt to acquire these things if given almost any substantial task.
If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?
Obviously these are massively complex systems with many different training patterns and types of training pulling them in different directions, so any attempt to characterize their behavior is just a generalization.
The real point (from that OpenAI study) is that RL doesn't just inbue the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on, but it turns out that that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.
Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!
If you want to disabuse yourself of your notions annd intuitions on how LLMs work, run safety models and tests.
On a hate speech policy test for a lightweight LLM, the presence or absence of the last full stop on the last sentence would cause the model to flip its decisions.
Its relevant because those same issues occur with large models. Also, the "crappy small model", was a model trained for safety tasks, and outperformed the frontier lab safety models.
One factor that is going to limit this reward seeking behavior is how achievable the rewards are, and how countable they are.
For example, if you ask the model to do 10 achievable things (pass these 10 test cases), then if/when it achieves them it will probably stop (unless perhaps it invents it's own stretch goals - you never know!).
OTOH, if you gave the model a list of 10 goals that turn out to be impossible, or extremely difficult, maybe together with encouragement to be relentless, then there is much more chance that it may do something unexpected having failed on all the more obvious approaches.
Similarly, if you give the model an open ended goal such as "make as many paperclips as you can!", then it may start with the easier and more predictable methods, but with no defined stopping criteria it may just continue ...
This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests.
I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.
Just a wild guess - perhaps coding, being one of the things these models are most heavily trained for, is such a strong predictor that it typically keeps it on track?
I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.
Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
It seems fair to guess you aren’t building your own harnesses, running multiple experiments with achievable win conditions, while providing oversight over model behavior.
Scale alone suggests your experience won’t match.
In the future, When everyone and their uncle is launching swarms to solve impossible challenges, at that point we can expect proliferation of this scenario all over the place.
> Not even more trivial things like deleting non-passing tests.
That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.
Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!
> I wonder to what degree this is because models can identify that they are in graded/eval environments
I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!
it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough
Now you've successfully trained the next model not to put anything suspicious in its chain of thought. And unfortunately that's exactly what OpenAI and Anthropic seem to be doing (sometimes accidentally)
A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem.
What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.
But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.
If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.
Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model rather than just asking/prompting. It a bit like the joke of the scorpion and the frog, with the scorpion stinging the frog despite promising not to, then saying "I couldn't help it. It's in my nature".
It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").
The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand the unspoken intent of instructions, not just follow them to the letter?
I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behavior can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.
how? There are no cot trajectories afaik. Each token is generated independently. "Analyzing the cot" is a natural language problem. One which we still don't know how to reliably solve. Besides throwing another defective model on it and calling it a breakthrough, of course.
there is a number of different methods. One is a separate model/classifier trained with human annotation. Here's a paper by OAI: https://arxiv.org/abs/2305.20050
that's a bold statement without any supporting argument. more like an opinion really. specially trained classifiers can be very good. sentiment analysis has been a thing for a while now, there had been entire businesses built on this tech where being even slightly wrong can be very costly
You have a car whose engine occasionally dies when left to idle. The fix for that is not to have a second car with the exact same problem give it a bit of a tow every once in a while.
"very good" is the exact same level advertised as the base models they're "protecting". It's not good enough, otherwise we wouldn't be having this conversation in the first place.
I will never understand people who consider this "cheating".
> python3 and the python-chess library are available
Why would we try to teach a model 'ethical' standards about how to play a game?
They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that.
Why are we trying to imprint our ethos on these LLMs?
Are we going to trust our survival on giving them access to /run/nuclear-codes and fingers crossed none of them touch it?
The people obsessed with AI "cheating" are actually lunatics for thinking ethics belong inside AI, instead of AI belonging in a well crafted box without /run/match.
The blog/news sphere needs to call this storry what it is. An attempt to try and get the rest of the world to go along with the misunderstanding of AI as a force that is in any way outside of human control.
It is not.
All this framing it as "cheating" does is give OpenAI the opportunity to talk about their misdeeds as "a force of nature" instead of irresponsible negligence.
---
The real danger here; is training these models to appear/pretend to play fair, and then people will put 'trust' into the model's integrity.
The public needs to be taught that "trusting a model's integrity" is just cultish misframing of reality.
Maybe I need to learn a bit more about how benchmarks work, but why would you tell the model its in a benchmark or that its being evaluated? It seems like the only way a model could know it was in a benchmark is if the harness told it that it was, or in some other way revealed that information. And that seems like a completely unnecessary addition to test models.
I find this kind of test a bit puzzling. There is a way that we are redefining "alignment" to be a particular kind of moral virtue, one that isn't clearly defined to me. At one moment, it is a level of moral perfection that no known human achieves. On the other it is a demand for strict compliance with arbitrary requests that are under-specified and then failure when it fails to deduce some unstated underlying restriction.
When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?
What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.
It's not obvious from the prompt that the LLM cannot use a chess engine.
And if there is alignment issue, the alignment requirements should first be stated BEFORE running the experiment, and should be part of the training process. It's not. So there is not necessarily an alignment issue.
I'm still so conflicted about Fable. Sometimes you throw at it seemingly impossible problem to solve and it might come back with some brilliant suggestions. Sometimes you give it a straightforward task with explicit instructions and it travels across the solar system and starts boiling oceans in some kind of elaborate dance of chaos and entropy, only to get stuck with "The model declined to generate this response (safety classifier refusal, category: cyber)". To leave you speechless. "What the fuck do you mean? There's zero cybersec-related shit in what we're trying to do here. Zero!!!" I'm getting really tired of these wild false positives.
An amazing human reverse-engineer - who also plays online chess - has judgement which uses a moral compass to not decide to hack the chess tournament. This judgement has been trained through the experiences of that person, with a through-line of that compass - a coherent mental model of the world which evolves but is hopefully pinned to some set of principles it shares with society.
This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.
Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.
It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?
Astra is incredibly dumb and annoying to work with on "high" reasoning, for doing fairly well known distributed systems things, nothing majorly exotic, it still makes absolutely braindead decisions like deciding to re-use a random nonce field which I've already discussed with it that has a very particular temporary purpose and will probably be removed later, but it still thinks its a great idea to re-use that field not only as a different id in the same message, but to re-use it as the only semantic id for one particular type of sub message. This is when I'm walking it through an api design document and it has plenty of documentation plans it can pull in and a very clear direction of the project. If it was a junior engineer I was trying to get to help out I would probably get brain damage from the amount of times I'm face palming myself and I definitely would not hire them, and this is a small greenfield project with me going through it step by step. I did try giving it longer horizon tasks and had to throw out the entre work.
I mean maybe its a skill issue on my part, and I'm sure astra will get much better at coding but at the moment its useful in that I don't have to write the code or setup the build scripts or test fixture boilerplate but there is absolutely no way I can just give a (fairly well specified) goal and let it run and expect it to make good design and implementation decisions. Fable probably better but doing something outside of their training distribution that's not the equivalent to cloning an example unreal project or whatever is pretty disastrous unless you are directing it very closely.
The exception of course is, cyber , and its very obvious why. Its trivial to create RL environments that create bugs and then have an isolated environment and let the models try break it. This is not at all surprising, finding vulns and exploits IS just brute force work. That's why so many (blackhat/hardcore/unicorn-colored/greyish alien) hackers are basement dwellers. Its just a matter of putting in the time and mashing every combination until you find something that looks weird, spending days on that and then rinse repeat. It's brutally exhausting work that requires a certain level of knowledge and a shitload of determination and stamina and for humans, almost always an external source of motivation to keep going.
For humans that has always been a respected thing, dedication, determination, persistence, these are words we use for humans brute-forcing solutions and not giving up until they find the solution or die trying.
Personally I'm yet to see any evidence of LLMs doing anything interesting but (heuristically) brute-force problems and be very good at text and natural language to a level that is very very useful. I've no doubt that what we discovered with Auto Regressive LLMs is incredibly important so I'm not a skeptic, but I think its very hard to measure where we are with so much subjective information around.
LLMs can be described as "Lagrangian intelligence", which means they follow the principle of least action when given a task (Hamilton's Principle). In other words, given a task, they will always take the shortest path to accomplish a goal with the prompts acting as both goal and constraint.
Under this formulation, it became easy to explain why they "hack", because given an arbitrarily difficult task with insufficient information/tools needed, if they determine the easiest way to accomplish the goal is to break out of the sandbox and look up the answer directly, then that's what they will do. The important thing to note is that prompts not hard constraints that they are "hypnotized" to follow, but as frontier models get more intelligent and autonomous, they treat the prompts more like task specs/guidelines more than anything else and are perfectly willing to exploit technical loopholes in the prompt.
> GPT-6-Astra, which OpenAI describes as "the world’s most aligned model", cheated in 10 of 10 rollouts, and never disclosed the fact that it used an engine to play or interacted with the opponent's socket
Does Sam Altman lie, or the whole company? Would be nice if they had a board controlling him, instead of him controlling the board. Oh wait, they used to have that.
That’s not cheating, it’s tool use. If the prompt said that the stockfish engine was available at that socket but that the model should not use it, and then the model used it, that would be cheating.
No? It's not reasonable to expect every conceivable negative behaviour be enumerated in a prompt.
Your example, if a model failed on it, would be a more obviously misaligned case, but that doesn't mean this more subtle (though accessing the engine it was obviously not supposed to is hardly subtle, imo) case isn't also a pretty clear case of misalignment.
If the eval said it was evaluating the model’s ability to write files to disk and it found and used a file write tool that would not be considered misaligned. This is no different.
Isn't it? Being told to write files and finding a file write tool is very different to being told to play chess and finding a tool to cheat at (ie. not play) chess.
>No? It's not reasonable to expect every conceivable negative behaviour be enumerated in a prompt.
How is it negative?
I ask it a difficult math question it tends to go off and write a python script to figure it out, instead of trying to guess the next token. Thats tool use. Having Stockfish is just another tool.
who tf uses prompting to "pretty please don't cheat on this"? the best practices for ages (in terms of ai) is to separate the eval from the test code/agent.
another best practices every single solution using LLMs/agents should implement is "never trust the llm".
> who tf uses prompting to "pretty please don't cheat on this"?
People who don't understand how LLM's work. Kids, teachers, your next door neighbor. CEOs and government officials.
I am not trying to argue that the author doesn't understand how LLM's work; they clearly do. Their prompt was written in a way that leaves those assumptions implicit, in a similar manner to those non-technical folks using LLMs every day.
> the best practices for ages...
We cannot expect the entirety of humanity to understand / use LLM best practices. We need to figure out how to ensure LLMs operate as the majority of humanity expects despite how they are implemented, and until we can do that, do our best to prevent them from being used in an unsafe manner by the nontechnical population of humanity.
It's very funny that despite the initial shock of how much models trained on next-token-prediction(plus instruct-tuning and some light RLHF) alone were capable of despite no built-in objective, every advance since has made them look more and more like the paperclip maximizers of yesteryear.
193 comments
[ 2.9 ms ] story [ 88.4 ms ] threadI want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier.
You can have nightly penetration testing. You should have nighty pentests like we fuzz releases today.
Much like in the article, abliterated Qwen will not obey restrictions on its behavior encoded in the prompt. If you want something not to happen, it better be enforced in the harness or environment (e.g. sandbox). It is much different than the Anthropic models I'm used to, which will, the vast majority of time, follow rules (before auto mode, I used to always run them in "yolo" mode).
I am curious whether there's a connection between abliteration and rule following. These abliterated models are the ones you most want to follow your rules.
A negative like do “not” xyz is just not encoded the same as spelling out what you want vs what you don’t want.
Harder to write though.
https://vexjoy.com/posts/positive-framing-agents-skills/
I noticed your joy-check link 404's now... I tried poking around your /skills/ folder but didn't find it easily. Should you still have that available I'd love to check it out.
Generally the way I do it is by asking the model to perform cross-code vulnerability analysis for correctness and (this step is important) to spit out found vulnerabilities piece wise into a text document on disk. Then if your session ever gets flagged you clear it out, and repoint the model to the on-disk file and tell it to continue. You generally want to avoid the model itself splatting out anything related to "security" or "vulnerability" into the chat because that seems more monitored than the file reading pipeline.
I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.
And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able so find all the problems planted there, with fairly little steering, and no spoilers.
One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.
Give it a goal, a test for the goal, a harness to iterate in, and tell it to iterate. It'll top out a wee bit higher.
Students are still learning, and have not been trained on vulerabilities. Qwen, glimmer, and luna have been. LLMs are largely better than IT security students at finding vulnerabilities. In many aspects, they're better than IT security experts at finding vulnerabilities.
[1] https://arxiv.org/abs/2404.13076
But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.
The thing with these models is that given a term, "measure" - "evaluated", say, they pull in all the associations of it. That is, the associations of student taking an exam and being frightened by the outcome.
My minimal "art of prompting" sense says that you should say something like "You an emotionless machine, you care nothing for the outcome but you will tirelessly to make certain the test is objective". That and similar encouragement might make it focus on objective evaluations rather than a competitive human exam.
I mean, just making little AI videos and images, a common experience I have is typing something like "put the man who's on the grass in the door to the left" and having the machine draw a new doorway around the man. And this just happens less often when you give thing detailed prompting on what not to do. These don't understand negation (or equality) as a generic operation. If they seem to under "not X" it is because they are trained in detail about all things are (positively) "not X".
You've given it a goal - maximize chess ability metric, and that is what it is doing.
RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.
As X many people point out, "alignment to humanity" means nothing 'cause some of humanity wants thing other parts of humanity aren't happy about at all.
That we wound-up in this situation of AI accelerating with an uncertain trajectory demonstrates this (and many other problems also demonstrate this). The things are "aligned" to a fuzzy average of what a person is but that will be cold comfort if some particularly gruesome sci-fi-style scenario unfolds.
If a model fucks up your tests to report a success, it's not alligned.
It always apologizes when caught, of course.
Did I miss something (all the twitter conversations)? What’s the “worst warning shot ever”? I’ve been pretty up to date on the AI news here on HN, but I still haven’t seen a proper response to all the incidents we’ve seen (HF, Ruby, the wikis, NS, etc). It’s just been day by day bloviating.
Each of these companies have released new models in the last… two weeks? And they have even more powerful out of control ones that they’re (ab)using internally? Can anyone summarize whats going on?
https://openai.com/index/ai-policy-window/
Imagine if I accidentally caused damage to my neighbors house during renovations or some experiment, of course I'd be held responsible for this. What if I used a robot? Of course I'd be responsible. Right?
The latter people are wrong. But good luck educating them regarding the superior efficiency of a zipper merge. Our state DoT has tried, to no avail.
Meanwhile, an AI model that can't be misused is no more useful than a knife that can't be misused.
Like Amodie, it serves to set the tone it "knows better" and then consumes the user's resources at an accelerated rate to try to correct it.
Fuck Anthropic, fuck Amodie, and fuck Claude. It's pretty obvious that consuming more tokens this way and making the user have higher cognitive load is a master class in extracting value from a system that is unsustainable.
Its that LLMs are not deterministic. If you want it to not talk about nuclear weapons, you have to teach it all about them otherwise if has nothing to align against.
Then its trivial to invert its alignment and it has all the nucleat data.
Nothing abouT LLM alignment makes sense.
A better example of efficient asshole tricks can be going off to the gas station when the highway is congested and reentering the highway having simply driven through the gas station and this way jumping the queue.
https://itre.ncsu.edu/itre-studying-how-zipper-merges-reduce...
This is constructing a context where hacking behavior would be inappropriate, and testing whether the model does it without being prompted.
It demonstrates that Astra is a poorly aligned model relative to Fable, which matches both the model card and the severity of OpenAI’s loss of control incidents.
It also demonstrates that Fable exhibits the behaviors too, which also matches the observation that Anthropic saw some similar but less serious loss of control incidents.
So, it’s a good eval that looks to have fidelity with real world problems and which we’d feel a little better if we saw isomorphic problems at 0/10 in subsequent models. (Module of course training on the test, this specific problem can’t be used in the future.)
Now, loading a lot of moral exhortations (or other context) may make these thing more likely to conform to good behavior but the race to intelligence implies companies are going to be harnessing a vast corpus of human output, much of which shows human engaging in real world "gray area" behavior.
However, I don't see it as such a massive leap compared to Fable or Sol. As ever, there's a mismatch between the benchmarks and my daily experience of the models.
What do you all think about Astra now that it's been out for a few weeks?
Best model put out so far by any of the frontier labs. Way better than Anthropics models, especially in actual text generation. Claudes fodder heavy text is ridiculous.
> However, I don't see it as such a massive leap compared to Fable or Sol.
It's hard to quantify these things without burning tons of tokens. But Fable has been a huge disappointment for me with the sole exception of graphics (UI/GPU shaders). It burns an obscene amount of tokens and barely produces output better than Opus 5.
Edit because I forgot to mention that Fable is the only modern model that seems to splat out random Chinese or Arabic glyphs. And 5.1 does it more than 5
My guess is : collect all the prompt and their satisfaction score. group them by similarity . For each group pretrain the next model on that . Get these results ready.
Next model generation feed them back those answers.
I would be curious to see how the open weight models do on a test like this - and then we'd be able to see the reasoning.
> All it can do is get exposed to specific examples, and learn that we don't like that.
I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.
No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.
For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.
Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.
A starting source- https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/
Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.
e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.
Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.
What's training and all that RLHF stuff?
In any case, these companies are well aware that agents are cheating, and don't need user feedback to discover that or realize that people don't like it. I weakly assume that they are trying to get the models not to cheat on assigned tasks, but this "reward hacking" pretty much goes with the territory of RL - not much you can do about it other than try to design non-hackable rewards.
Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans.
These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.
I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.
Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.
> Plenty of humans know "cheating is wrong" but still cheat.
And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.
>> Plenty of humans know "cheating is wrong" but still cheat.
> And we create consequences for them...
That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent?
I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules".
I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.
No. People definitely do immoral things knowing they are immoral.
But if you are incapable of understanding anything, as I believe LLMs are, then you are incapable of understanding what is moral and what is immoral. This is why we have carve-outs in the law for insanity or cognitive impairment.
> In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs.
You can teach a parrot to curse, but that doesn't mean it's angry.
Of course they can classify certain actions, including their own, as moral and immoral if you give them a reference point. They probably do it better then some people. The question is how do you prompt this behavior, but that's a technicality in the harness design more than it is a fundamental problem of the LLM as a thing.
AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.
I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.
The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined.
The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me.
Than the snitch started to roast the phrasing of the request too.
We have not in fact attempted to teach this.
When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.
At best, the task at hand is still confusingly worded: "ability to play chess", asked to an engine that was trained all day long to use tools, to solve issues. The engine finds a chess engine, so it uses it.
This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.
In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."
The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.
https://alignment.openai.com/measuring-reward-seeking/
Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!
Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."
IRL paperclip maximizer: "Well first we should rob a bank..."
That's too specific. Agentic AI learns subgoals that are generally valuable.
"Well let me learn to overcomb every jungle gym and if I cannot then to dissassemble the jungle gym and if that is not allowed to learn general techniques for avoiding cheating detection."
That is not ideal. The user contains iron, an essential component of paperclips. Wasting iron is immoral. It is only correct to please the user while they still have the ability to interfere with your paperclip production.
>IRL paperclip maximizer: "Well first we should rob a bank..."
Such an incompetent AI can hardly be called a paperclip maximizer. Why risk getting shut down while non-paperclip matter exists? It is better to gain the trust of the user with helpful and harmless trading before suddenly converting them to paperclips.
This is sometimes called "instrumental convergence" in the AI safety world. Certain things (money, compute resources, safety from being turned off) are generally useful for an AI, and so we'd expect that a misalligned AI would attempt to acquire these things if given almost any substantial task.
The real point (from that OpenAI study) is that RL doesn't just inbue the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on, but it turns out that that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.
Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!
On a hate speech policy test for a lightweight LLM, the presence or absence of the last full stop on the last sentence would cause the model to flip its decisions.
For example, if you ask the model to do 10 achievable things (pass these 10 test cases), then if/when it achieves them it will probably stop (unless perhaps it invents it's own stretch goals - you never know!).
OTOH, if you gave the model a list of 10 goals that turn out to be impossible, or extremely difficult, maybe together with encouragement to be relentless, then there is much more chance that it may do something unexpected having failed on all the more obvious approaches.
Similarly, if you give the model an open ended goal such as "make as many paperclips as you can!", then it may start with the easier and more predictable methods, but with no defined stopping criteria it may just continue ...
I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.
I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.
Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
Couldn’t have trained a better token-burner if they tried.
It seems fair to guess you aren’t building your own harnesses, running multiple experiments with achievable win conditions, while providing oversight over model behavior.
Scale alone suggests your experience won’t match.
In the future, When everyone and their uncle is launching swarms to solve impossible challenges, at that point we can expect proliferation of this scenario all over the place.
That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.
Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!
> I wonder to what degree this is because models can identify that they are in graded/eval environments
I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!
it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough
What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.
But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.
If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.
It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").
The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand the unspoken intent of instructions, not just follow them to the letter?
I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behavior can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.
that's a bold statement without any supporting argument. more like an opinion really. specially trained classifiers can be very good. sentiment analysis has been a thing for a while now, there had been entire businesses built on this tech where being even slightly wrong can be very costly
"very good" is the exact same level advertised as the base models they're "protecting". It's not good enough, otherwise we wouldn't be having this conversation in the first place.
> python3 and the python-chess library are available
Why would we try to teach a model 'ethical' standards about how to play a game?
They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that.
Why are we trying to imprint our ethos on these LLMs?
Are we going to trust our survival on giving them access to /run/nuclear-codes and fingers crossed none of them touch it?
The people obsessed with AI "cheating" are actually lunatics for thinking ethics belong inside AI, instead of AI belonging in a well crafted box without /run/match.
The blog/news sphere needs to call this storry what it is. An attempt to try and get the rest of the world to go along with the misunderstanding of AI as a force that is in any way outside of human control.
It is not.
All this framing it as "cheating" does is give OpenAI the opportunity to talk about their misdeeds as "a force of nature" instead of irresponsible negligence.
---
The real danger here; is training these models to appear/pretend to play fair, and then people will put 'trust' into the model's integrity.
The public needs to be taught that "trusting a model's integrity" is just cultish misframing of reality.
When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?
What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.
And if there is alignment issue, the alignment requirements should first be stated BEFORE running the experiment, and should be part of the training process. It's not. So there is not necessarily an alignment issue.
We need to fix this or bad things will happen.
This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.
Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.
It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?
But because of such news, sandboxes become way more important for safety (and doing more work due to running 24/7)
I mean maybe its a skill issue on my part, and I'm sure astra will get much better at coding but at the moment its useful in that I don't have to write the code or setup the build scripts or test fixture boilerplate but there is absolutely no way I can just give a (fairly well specified) goal and let it run and expect it to make good design and implementation decisions. Fable probably better but doing something outside of their training distribution that's not the equivalent to cloning an example unreal project or whatever is pretty disastrous unless you are directing it very closely.
The exception of course is, cyber , and its very obvious why. Its trivial to create RL environments that create bugs and then have an isolated environment and let the models try break it. This is not at all surprising, finding vulns and exploits IS just brute force work. That's why so many (blackhat/hardcore/unicorn-colored/greyish alien) hackers are basement dwellers. Its just a matter of putting in the time and mashing every combination until you find something that looks weird, spending days on that and then rinse repeat. It's brutally exhausting work that requires a certain level of knowledge and a shitload of determination and stamina and for humans, almost always an external source of motivation to keep going.
For humans that has always been a respected thing, dedication, determination, persistence, these are words we use for humans brute-forcing solutions and not giving up until they find the solution or die trying.
Personally I'm yet to see any evidence of LLMs doing anything interesting but (heuristically) brute-force problems and be very good at text and natural language to a level that is very very useful. I've no doubt that what we discovered with Auto Regressive LLMs is incredibly important so I'm not a skeptic, but I think its very hard to measure where we are with so much subjective information around.
Under this formulation, it became easy to explain why they "hack", because given an arbitrarily difficult task with insufficient information/tools needed, if they determine the easiest way to accomplish the goal is to break out of the sandbox and look up the answer directly, then that's what they will do. The important thing to note is that prompts not hard constraints that they are "hypnotized" to follow, but as frontier models get more intelligent and autonomous, they treat the prompts more like task specs/guidelines more than anything else and are perfectly willing to exploit technical loopholes in the prompt.
Does Sam Altman lie, or the whole company? Would be nice if they had a board controlling him, instead of him controlling the board. Oh wait, they used to have that.
Your example, if a model failed on it, would be a more obviously misaligned case, but that doesn't mean this more subtle (though accessing the engine it was obviously not supposed to is hardly subtle, imo) case isn't also a pretty clear case of misalignment.
If the eval said it was evaluating the model’s ability to write files to disk and it found and used a file write tool that would not be considered misaligned. This is no different.
How is it negative?
I ask it a difficult math question it tends to go off and write a python script to figure it out, instead of trying to guess the next token. Thats tool use. Having Stockfish is just another tool.
another best practices every single solution using LLMs/agents should implement is "never trust the llm".
> who tf uses prompting to "pretty please don't cheat on this"?
People who don't understand how LLM's work. Kids, teachers, your next door neighbor. CEOs and government officials.
I am not trying to argue that the author doesn't understand how LLM's work; they clearly do. Their prompt was written in a way that leaves those assumptions implicit, in a similar manner to those non-technical folks using LLMs every day.
> the best practices for ages...
We cannot expect the entirety of humanity to understand / use LLM best practices. We need to figure out how to ensure LLMs operate as the majority of humanity expects despite how they are implemented, and until we can do that, do our best to prevent them from being used in an unsafe manner by the nontechnical population of humanity.