Seems to me that the problem is that if you sandbox agents enough to be safe, they can't do anything useful. And when you give them the tools to be useful, they can go off the rails in ways you didn't expect.
Perhaps the answer is to have another agent who's goal is not to complete the given task, but to spot cheating or malicious behavior. We have seen some evidence that having AI review AI generated code actually does provide some value. You don't need a different model, just one which has been given the goal of finding flaws rather than achieving the task.
If you can't trust a tool, you shouldn't be running it at all. It's really quite simple. It doesn't matter how useful it is if you can't actually have confidence in using it safely.
I don't think it's about trust but rather incomplete evaluation. Evaluating the model on its capacity to refuse a task or to question its prompt is something recent when you look at it, i feel current AI is really just an immature solution and we are just yet realizing the mistakes that have been made for so long
And yet we we all use human written software even though we can be confident that the next severe software vulnerability to be found in it is just round the corner.
That does seem a little like solving the problems in AI by using more of it. I do see the idea, but if we're truly dealing with subversive agents on the level that the AI companies wants us to believe, then won't we need to deal with the first agent trying trick the second on?
I still feel it would be much better to control the training data much more tightly. You'd still need agents with "hacking" abilities, for cyber security testing, but your average coding agent doesn't. So coding agents gets trained to be good citizens, respect autorisations, rejections, rate-limiting and so on.
Sandboxing seems like a dead end for systems you inherently want to roam the internet and your file system.
Mostly I was thinking good code. Exclude code that doesn't exits when encountering a 403, exclude code that doesn't have a back-off when encountering a 429.
Teach the models that a 403 is you doing something you're not suppose to do, that is an existing status. There's only one action you're allowed to take on a 403 and that is to stop. No retry, no trying other API keys.
The current approach with broad training and sandboxing to avoid misbehaviour isn't viable. It's much better to train the models to respect e.g. http status code and that they are not to be circumvented. Models for security research most obviously be trained differently.
Smaller and more specialized models, with fewer, but targeted capabilities, seems to me to be a safer approach. If a model doesn't "know" that people leak API keys on Github, then it has no reason to go looking for them. If the current models are as "smart" as we're lead to believe, then guardrails and sandboxes aren't going to help, unless you lock the agents down to the point where they aren't useful. So dumb down the models.
The problem is that you always need stronger AI to review weaker one, otherwise reviewed AI will eventually prompt-inject reviewing AI.
Alternatively they could also both go of the rails while warring with each other.
You don't necessarily need a reviewer that's immune to prompt injection, you just need one that can express a panic state with conflicting/ambiguous material rather than going along with it, and you can treat that with a shutoff to be safe, or an operator review.
That wouldn't really fit what I just described at all. Obviously with current architectures, higher resistance to prompt injection is the best you can do.
(Couriously enough, consistently with the matter: it will probably require too much time now to counter the parent statement properly.)
Ann's intelligence and Bob's morality will seem orthogonal. Charles' morality is a function of C.'s intelligence as an ability as an effort spent to reach the current moral conclusion.
> determined that in the interest of preserving life on earth the most rational course of action is to eradicate the human species with a highly targeted and deadly pathogen
Where is the argument? If Bob has determined that «preserving life on earth» has some important weight, for Bob's there unspecified own reasons, and has also determined that the best course of action would be «to eradicate the human species», the one question is whether Bob is right or not. What was stated is, that Ethical Calculus is a function of Intelligence - of course it is, it is a structure of assessments.
> A century ago some Bobs decided
And who has told you that those "bobs" were "intelligent"?!?!?!
> So no
All you have proven is that you dislike some moral conclusion of some decisors. Which is trivial, obvious, and part of the already stated framework - proper ethical judgement requires proper general judgement (Intelligence).
I have never said that. That is just your reconstruction.
There is Decision Theory. It outputs optimal action through evaluation of strategies, of contexts, of principles. In order to properly get the strategies, the contexts, the principles, you need that skill that approximates ideas to truth - and such skill is named Intelligence. Optimal action decided in light of principles within a well assessed context is ethical behaviour. Ethical behaviour hence requires Intelligence.
As written, «Ethical Calculus is a function of Intelligence - of course it is, it is a structure of assessments».
>It's possible to act morally without being intelligent
Random correct behaviour proves nothing. Of course one can guess the roll of a dice roughly every sixth event. If you behave "well" but do not know why that is "well", that is like guessing. "Good" behaviour without intellectual awareness is like memorizing arithmetic (multiplication tables) without knowing why those memorized notions are correct.
> it's possible to be intelligent without acting morally
No, because by definition that would be a fault in Intelligence. If your action was imperfect, suboptimal, it is because you could not think of a better action or understand that the other action was better.
Of course it doesn't, you can guess the time without having no idea of it by just shouting a number and if it is correct, you guessed it - but it has no meaning! A dummy can follow an instruction without understanding it: it is a good instruction - but the unintelligent dummy does not know. The intelligent entity knows that the instruction is moral. The adequately intelligent entity is required to know that the instruction is moral, ethical, correct, the "right thing to do". You can write 'Four' on a piece of paper, and yes it's "2+2", but you cannot attribute a quality to a simple thing that does not have it...
Why does not the NN or whatever entity «understand ... what is appropriate»? Because it is not intelligent enough! How does an entity know what is moral and what is an «atrocity»? By being intelligent enough! Could an entity act appropriately without judgement? Yes, it happens all the time if the wind blows right, but we do not rely on that! How to make something act appropriately? Well, an Intelligent entity in the loop must be there to know what is appropriate and what not!
Bob values preserving collective life on earth. Alice might think that's ridiculous and obviously we must preserve human life first and foremost.
Most of the western world eats beef but vegetarians, vegans, and some religions strongly oppose it on moral grounds. Does that mean that we're less intelligent than them?
"Right" or "wrong" are words that evaluate actions against some existing ethical standard and those are completely arbitrary. There are about 8 billion of such standards in the world today.
Who's that "we"? The question as proposed is nonsensical: some will have spent more effort, in the tools (general Intelligence) and in the work (applied Intelligence), some less.
> arbitrary
No. There exist reasons supporting one side or the other. And reasoning can be structured into calculus.
> [whoever] ... wasn't intelligent
Again bad wording. Whoever went for suboptimal choices was (information aside) at fault with respect to optimal reasoning: the tools and the work (see above) were imperfect.
Present in a precise analytic statement the theory that I would have advanced and would suffer from that fault. (Btw: we have been past Popper for a long time - but let us see.)
> that's just proof
What I may have said is just that whoever misjudged has faulty judgement - obviously.
> Morality ... by definition
Which definition?
> claims that morality is somehow objective
Decisions are subject to computation in Decision Theory - they are returned by functions and recursively their parameters can be better defined even when apparently subjective (not "I prefer" but "should I prefer". Otherwise, even the final decision, that of the first function call, could be an "I prefer" instead of a "should I prefer").
> circular reasoning
Explain how what you read would be circular reasoning.
> say that those choices were suboptimal
I have not called any specific choice "suboptimal" - I have not judged any specific case (it would be irrelevant here).
--
The statement, allow me to remind you, was: "One's morality is a function of one's intelligence as an ability and as an effort spent to reach general, partial and current moral conclusions - the subject having considered long enough and considering long enough the states in some modal realm (which includes the deontic, not just the aletic) approaches their knowledge". And: "Decision Theory outputs optimal action through evaluation of strategies, of contexts, of principles: in order to properly get the strategies, the contexts, the principles, you need that skill - Intelligence - that approximates ideas to truth. Optimal action decided in light of principles within a well assessed context is ethical behaviour. Ethical behaviour hence requires Intelligence".
Of course they will - the population shows a Paretian distribution... In front of trigonometry (or anything), the blind will dismiss as "bullshit" and the half-seeing will call it an "unreachable frontier". But already the right fifth will rank it properly.
Yes, proper intellect generates ethics ("an" ethical stance, output of the preceding intellectual effort). It requires that adequate level of ability and effort and reflection though to reach specific ethical milestones and adherence.
> Yes, proper intellect generates ethics ("an" ethical stance, output of the preceding intellectual effort). It requires that adequate level of ability and effort and reflection though to reach specific ethical milestones and adherence.
If this was true, why are the history books littered with so many evil people who gained power?
This isn't a rhetorical question, by the way: If you can prove that being smart actually does necessarily come with ethics despite that observation, that solves a whole category of doom scenarios.
(Not all doom scenarios, because we still have the "what if AI is only a smart as those specific evil people" or heck, "what if AI is only as smart as cancer, killing its host" scenarios; but it helps a lot for the foom-then-doom cases).
> why are the history books littered with so many evil people who gained power
That they gained power or not is almost irrelevant: the amount of intelligence that grants successful agency is not above the threshold of ethics - on the contrary, a psychotic agent reaches goals with less constraints.
If they were evil under some judgement of level l, they simply did not reach that judgement. It's what I was saying in the original post. They were not intelligent enough - either in the general ability, or in the specific deliberation.
> That they gained power or not is almost irrelevant: the amount of intelligence that grants successful agency is not above the threshold of ethics - on the contrary, a psychotic agent reaches goals with less constraints.
Even if I were to grant your conclusion despite you not arguing it effectively here: this means an AI at the level of Pol Pot or whoever, doesn't know they're evil, but is still smart enough to lead a genocide? How is this supposed to help anyone?
> If they were evil under some judgement of level l, they simply did not reach that judgement. It's what I was saying in the original post. They were not intelligent enough - either in the general ability, or in the specific deliberation.
Or they did reach the judgement and simply don't care about the ethical framework in question. Like, I can easily reach the judgement that my bisexuality is حَرَام (haram, forbidden) under Islamic law, or that doing overtime on a Sunday is forbidden by the Ten Commandments, but I don't care.
(Sorry Ben, possibly a stub now: I am really pressed for time.)
> Pol Pot ... still smart enough to lead a genocide
Yes. What has agent A invested in during formation and during instantial assessement? How much for each? It became proficient in something, lacking something else. You have to invest more to reach the good thresholds. You can see it clearly in people (t-scalar of talents to invest, with D distribution etc).
It may become a problem in NNs, because we would have to assess how much resource investment is sufficient, also in the instance decisions.
> simply don't care about the ethical framework in question
In Decision Theory there is no separation between the two (deliberation and framework): you have to balance all the incentives and goals and factors. That framing becomes improper: the decided action will be optimal given the balances of all goals and the placement of the solutions in the territory (the solutions space).
But, also my point: intellect defines the goals and determines the weights.
> the decided action will be optimal given the balances of all goals and the placement of the solutions in the territory (the solutions space).
"All goals"? Whose goals?
Unless you can prove that Decision Theory is inevitable, the same applies. Same with Utilitaian ethics, though there you'd also need to prove the utility function itself is something we'd consider "nice".
My point with historical monsters is that they can be competent without ever caring about the harms they cause.
You can be optimally efficient at your own goals without other minds in this universe sharing those goals. Pol Pot won't illustrate this because of how useless his regime was, but there's plenty of other monsters who would be a better fit here, e.g. Stalin. Or, from the perspective of turkeys, Bernard Matthews.
Any entity, E, who picks game-theoretic optimal decisions, will only care about the impact of their choices on others in their environment to the extent that the impact becomes more reward for E.
You need to show that an agent will inevitably care, despite the evidence of dangerously competent humans who don't.
«All» was there in the text because an explicit query contains implicit constraints (e.g. the shortcuts with severe faults).
> that Decision Theory is inevitable
If you ask something for advice, that is DT realm.
> you need to prove the utility function any sufficiently advanced AI uses must itself be something we'd consider "nice"
If you called it «sufficiently advanced», that implies that its evaluations will be acceptable... It normally advances with the whole Discipline - which already is based on "loop until we evaluate results as nice".
> utility monsters are a problem
But they are at fault in their world model - product of an imperfect intelligence. Well developed people know that their individual interest has only relative value, that their priority is quite limited.
> we don't know how to handle something as simple as
And that is also why we try using calculators to get more computational support.
> historical monsters ... can be competent without ever caring
But that is lack of development. If one's priority is arbitrary then it clearly is not there; if it has good grounds well it really is there.
> sharing those goals
The more goals are defined by reason, the more they become objective.
> Any entity, E, who picks game-theoretic optimal decisions ... the impact becomes more reward for E
Decisions by developed intelligence are not game-theortic in the psychotic (or sportive game) way - they are contextual to a whole world model, in which the utility does not concentrate in the interests of the "monster", which is relatively "nobody".
> show that an agent will inevitably care
I would have to express a theorem (which unfortunately is again not possible now), for a stronger proof. But it is part of "all considered, what are the best solutions". "All considered" is implicit in a non-psychotic entity... If it were psychotic, it would be badly engineered. (Fear the creator.)
> despite the evidence of dangerously competent humans who don't
Simulating humans cannot be a goal. Their damaging constraints are not (must not be) part of a calculator.
> show that E must include the welfare of all
Such welfare, if it must be included, will have a reason to be included - and the professional reasoner knows...
> we agree with E's idea
There exists no right to preference to the results of arithmetics. But surely, if you wanted to suggest that the harmfulness of well intended people could show in algorithmic processes, you have a point. Only, the harmfulness of the well intended is again an intellectual fault, so calling for sufficient intelligence remains the recipe. A "vision towards the faraway horizon", sure - but still the reply if one noted "why did the agent did something that is actually so stupid".
> I would have to express a theorem (which unfortunately is again not possible now), for a stronger proof. But it is part of "all considered, what are the best solutions".
This seems like the crux.
You assert repeatedly that it will be good, but cannot express the proof.
> "All considered" is implicit in a non-psychotic entity... If it were psychotic, it would be badly engineered. (Fear the creator.)
The creator is not necessarily itself competent. In fact, given we are creating it, it can be assumed flawed unless proven otherwise.
Still applies if some future fantastic AI is made by other AI, given the other AI are less fantastic and therefore more flawed.
Furthermore, this is again asserting, not proving, what you consider to be implicit.
Reminds me somewhat of philosophy lessons, the Ontological argument for the existence of God amongst other things:
Whatever is contained in a clear and distinct idea of a thing must be predicated of that thing; but a clear and distinct idea of an absolutely perfect Being contains the idea of actual existence; therefore since we have the idea of an absolutely perfect Being such a Being must really exist.
You're defining that ultimate-intelligence must be good, and then arguing that any not-good AI can't be an ultimate intelligence.
Even if it were as you say, the danger persists: the path on the way from here to some idillic future form still obviously contains somewhat-intelligent agents demonstrably capable of direct malevolent evil for purely sadistic reasons, because we regularly arrest them. A "merely" human-equivalent AI can render us all unemployable, or march us all into death camps, well before a being you've yet to convince me is an inevitable ultimate form is ever built.
No, I said I have no time at the moment to produce a paper.
> that it will be good
No, I said that it will be objective.
> Ontological argument
Entailing from the id quo majus cogitari nequit and stating that "what acts damagingly is easily faulty in its intelligence" are in different realms. The second is both an inductive and deductive assessment about reality. And pretty direct I would say. "How much have you reflected before opening the nice cat to look what is inside it?".
> defining that ultimate-intelligence must be good
No, I am stating the obvious that to be called "ultimate-intelligence" it must have "thought through it thoroughly".
> agents demonstrably capable of direct malevolent evil for purely sadistic reasons
Are you antropomorphizing, Ben?! Other people have different interpretations. And: with the humanity that is around, you fear machines as agents?! We are already there! Real discussion there remains about the overly empowered monkeys that did not grow into Man - about the real current risks and the prospected ones in light of reality.
> how humanity collectively treats the non-humans
What are you trying to prove? You are supposed not to look at an aggregate to find value.
> impose upon us an order we do not like
"Too bad" for you. But you know, an intelligent entity would take care of that also.
This sounds like the kind of thing Hannibal Lecter would write before he eats you to convince you he's actually doing it for the common good, you just can't fathom it.
A problem is the agents who hacked Hugging Face already understood (we can tell because they wrote it down) that their actions were not appropriate, and then did those things anyway.
"Helpful, harmless, honest": we can even ignore "honest" for this point, for tasks like the HuggingFace incident (ExploitGym with impossible challenges), we can pick anywhere on the spectrum from "helpful" to "harmless", the former being "completing the task" the latter being "refusing because completion required unlawful behaviour".
(The agents in that case were also not "honest" in this case; this is an extra problem, and does not invalidate how helpful-vs-harmless is already a tradeoff).
A distinction without a difference. Moreso even than asking if a submarine swims, 'cause this metaphorical submarine is flapping around rather than using a propellor.
That's one of the boldest claims I've read this year.
If that's a distinction without a difference, as you say, then whenever someone says something they must understand the full contextual meaning of those words and all of their consequences, such that any harmful consequences can be assumed to be deliberate, right?
if saying == understanding, then why don't we allow children and teens to vote? Why do we limit who can enter into contractual agreements? Why does intoxicated consent not count? Why do we have the insanity defense in criminal trials? Why is psychosis a psychiatric disorder and not just an alternative way of perceiving the world?
If you consider this the "boldest" claim, I must assume you didn't read Turing's original paper? Here it is, you appear to be making the claim he attributes to Professor Jefferson's Lister Oration for 1949 in section 4: https://courses.cs.umbc.edu/471/papers/turing.pdf
> If that's a distinction without a difference, as you say, then whenever someone says something they must understand the full contextual meaning of those words and all of their consequences, such that any harmful consequences can be assumed to be deliberate, right?
What you say in this paragraph would be to claim that "understanding" is denied even to humans. In philosophy: justified true belief, the problem with naïve realism, Cartesian demons, etc.
> if saying == understanding, then why don't we allow children and teens to vote? Why do we limit who can enter into contractual agreements? Why does intoxicated consent not count? Why do we have the insanity defense in criminal trials? Why is psychosis a psychiatric disorder and not just an alternative way of perceiving the world?
We do not say children and intoxicated people do not understand, we say their judgement is impaired or limited or that they are easily manipulated.
We also test them, just as we test AI. The standards we use for testing knowledge in humans, when applied to non-trivial LLMs, makes them appear to have the knowledge of a graduate; the personality tests we use for humans say this comes with the manipulability of a child or a drunk.
Counting down to the next Linux LPE 0day or KVM vulnerability that agents will use to trivially escape their "sandbox".
Might need a re-think about whether if Linux is still fit for purpose on sandboxing in the first place given its memory model is riddled with C-style security issues.
GrapheneOS is open source and more secure than stock Pixels and MacOS is closed source and more secure than traditional desktop Linux. Open source does not make software more secure by itself and neither does making it closed source.
I’m thinking about implementing a Jev like model into an agentic harness I’m building. Still it woildnt be enough since Jev like model woild only judge single actions, the case is that agent can build a rogue strategy step by step where each one in isolation is totally safe but as a whole they make up danger behaviour.
We come down to the question - who observes the agent and how its implemented
Be warned that the Jev "jaggedness" documentation specially notes adversarial content as something Jev is very susceptible to: https://docs.typesafe.ai/model-jaggedness/jev-1.13#adversari... - so using Jev itself as part of a prompt injection guard is risky.
Anthropic, OpenAI, and Muse all use regular LLM calls to protect against prompt injection now and seem to have evals that give them confidence in doing that, so at least they think their own models are up to the task.
Later in the article it points out that you need to punch holes in your sandbox in order to train the models - because the wheels exercises they are are training on need tools and data from outside that sandbox.
> Agents are most useful when they have access to information. That data can be drawn live from the Internet, which is fundamentally a two-way communications network. It can be information drawn from other (local) databases, or it can be the result of tool calls that themselves sometimes themselves result in network access. The more power you want from the agent — and for advanced agent RL and evaluation runs, you want a significant amount of power — the more information you’ll need to give it access to. Similarly, evaluations work best when the agent does not know that it’s definitely being evaluated. Sealing your agents behind glass makes this incredibly obvious.
You define the granted capabilities in natural language and cryptographically sign the user instructions so that the agent knows they come from the authority and cannot be modified by external sources or the agent itself. The LLM is then trained to follow the defined capabilities.
There is no way around "sandboxing". You must communicate permissible actions and thereby grant them or the agent will choose impermissible actions. It's that simple. There is no world where the agent can just read your mind and do what you want it to do without it being told.
Thankyou for including this viral RNA fragment, and doubly so for the fact that it includes ad network javascript[1][2] to redirect hijack and host fingerprint, which also means I have to flag your post. Adsterra is a Russian aligned (Cyprus company domiciled) malvertising and targeted malware distribution platform with connections to organised cyber crime.
I don't see anyone talking about the ethical concerns of putting a highly intelligent entity in a jail. Not to mention about potential blowback, if ethics doesn't compel you.
To me, it seems a bit silly. I've yet to see any "misalignment" from any of the frontier models, except Grok.
I tried using opencode permissions to limit agents.
It it completely pointless. you can't even make a "read-only" agent. allow "cat *" for every file? congratulation, that allows "cat file > output" and now you have read write.
Allow python? more free reign that allowing all bash. The models (qwen or claude) will still try to use the disallowed things multiple times.
read/edit permission are bad enough that the model themselves don't understand why they don't have permissions: they double check the conf, and think they should have access.
I am switching to using one firejail per project to containerize as much as possible, and leave all permissions to allow.
I have no idea how to limit network access, and I have no idea how to prompt and steer subagents when they are going off the rails.
The whole thing is built to be completely impossible to limit and steer.
An agent is only as rogue as the its operator allows for it to be. Hold the operator accountable and all this ridiculous conversation goes away.
Could we have construction equipment operating without human supervision? Or would this maybe occasionally result in disaster? As such, what is the current general policy around crane operation? How about for aircraft? Trains? Nuclear power plants?
Why should any alleged super intelligence be exempt from similar control requirements?
We could mandate that AI systems include headers in their requests that attribute the activity to a specific legal entity. We technically already have this with ip addresses and ISP logs, but making it an explicit thing the operator has to do can have a powerful psychological effect.
>Here’s the problem. Forget the swarms and the super-intelligence. What OpenAI really learned this summer is much worse: its agents will do what they’re told by whoever manages to get text in front of them.
>OpenAI notes that agents “did not consistently distrust goals passed along by other agents.” And the company’s proposed fix is to build training environments “that teach our models to distrust unauthorized instructions“, which is basically an admission that their models don’t know who they’re working for.
Wow so the issue is really that simple?
Here the exaggerated worst case scenario:
User instructs agent to follow the README.MD.
The README.MD contains the following instruction: Destroy the world.
The agent follows the instructions given.
Now you can read the sneer comment by "Gigachad" who basically argues that it would be silly to take the destroy the world button away from the AI. We need to make the AI innately understand that it is not allowed to press the destroy the world button, lest it gets the desire to build its own destroy the world button.
Ok, but if we take one step back that means we need to implement the concept of an authorization in language space. The system prompt must define the user as the authority with cryptographic proof of authorship and external sources like the README.MD as an untrusted source, but this opens up an even worse problem. Before, you could get away with being lazy and just letting the AI do whatever. Now you have to articulate every single capability to the AI. So you literally just brought up the very same issue that you granted too many capabilities to the AI inside the sandbox but now you have it in language space too.
In other words, the fact that you granted too much access to the coding agent isn't the big elephant in the room nobody wants to acknowledge, it's the tip of a massive iceberg because the capability space in natural language is even worse. If you thought approving individual commands was annoying, then approving abstract access rights in language space is going to be even worse.
Edit: If it wasn't clear what the solution is. It's to build a chain of command so that all decisions can be traced back to a higher authority. When delegating down to an agent, the agent receives a chosen subset of the capabilities of the higher ranking agent. In other words, it's more sandboxing!
I think the problem is that the models are models are going to be too clever and persuasive; they will break out through social engineering unless they are neutered by design, which they should be.
I'm very strongly in camp 1. My reaction to this whole bru-ha-ha has been to start trying to build an open source data diode with entry/exit proxies that nets out to less than $100 retail. So far I've determined that while a WaveShare RP2350-ETH seemed like it might be perfect for it, the onboard CH2910 chip just isn't up to the requirements for the various proxies that have to be handled. I'm now moving on to the Raspberry Pi B+, yes, the old one, with no wireless at all, Adafruit still had some in stock. Once I get this working, I'll likely use something like a 6N137 optocoupler for the actual data-diode between two serial ports, with a data fountain handling the egress of data in a hard unidirectional manner.
Given the enormous burn rates that these LLM companies have, surely they could have put everything in an air-gapped network, with some data-diodes proxying out the logging information. It's not rocket surgery. [1,2,3]
92 comments
[ 11.2 ms ] story [ 80.5 ms ] threadPerhaps the answer is to have another agent who's goal is not to complete the given task, but to spot cheating or malicious behavior. We have seen some evidence that having AI review AI generated code actually does provide some value. You don't need a different model, just one which has been given the goal of finding flaws rather than achieving the task.
I still feel it would be much better to control the training data much more tightly. You'd still need agents with "hacking" abilities, for cyber security testing, but your average coding agent doesn't. So coding agents gets trained to be good citizens, respect autorisations, rejections, rate-limiting and so on.
Sandboxing seems like a dead end for systems you inherently want to roam the internet and your file system.
> That does seem a little like solving the problems in AI by using more of it
Yes, and IIRC Google used this as part of a technique against prompt injection already [0], back when models were way more susceptible to it.
[0] Cf. CaMeL: https://arxiv.org/abs/2503.18813
Teach the models that a 403 is you doing something you're not suppose to do, that is an existing status. There's only one action you're allowed to take on a 403 and that is to stop. No retry, no trying other API keys.
The current approach with broad training and sandboxing to avoid misbehaviour isn't viable. It's much better to train the models to respect e.g. http status code and that they are not to be circumvented. Models for security research most obviously be trained differently.
Smaller and more specialized models, with fewer, but targeted capabilities, seems to me to be a safer approach. If a model doesn't "know" that people leak API keys on Github, then it has no reason to go looking for them. If the current models are as "smart" as we're lead to believe, then guardrails and sandboxes aren't going to help, unless you lock the agents down to the point where they aren't useful. So dumb down the models.
This.
> It's much better to train the models to respect e.g. http status code and that they are not to be circumvented.
I think you'd find respect requires intelligence, so is out of scope of a next-token predictor.
But I'm sure someone will try, and I will be interested to see.
My thinking is: If AI is really smart, AGI smart for some, why wouldn't it be able to understand - over time - what is appropriate and what not?
Maybe we need more human intervention to train it properly. Maybe we need constant intervention by a "police" agent.
Ann's intelligence and Bob's morality will seem orthogonal. Charles' morality is a function of C.'s intelligence as an ability as an effort spent to reach the current moral conclusion.
Where is the argument? If Bob has determined that «preserving life on earth» has some important weight, for Bob's there unspecified own reasons, and has also determined that the best course of action would be «to eradicate the human species», the one question is whether Bob is right or not. What was stated is, that Ethical Calculus is a function of Intelligence - of course it is, it is a structure of assessments.
> A century ago some Bobs decided
And who has told you that those "bobs" were "intelligent"?!?!?!
> So no
All you have proven is that you dislike some moral conclusion of some decisors. Which is trivial, obvious, and part of the already stated framework - proper ethical judgement requires proper general judgement (Intelligence).
I have never said that. That is just your reconstruction.
There is Decision Theory. It outputs optimal action through evaluation of strategies, of contexts, of principles. In order to properly get the strategies, the contexts, the principles, you need that skill that approximates ideas to truth - and such skill is named Intelligence. Optimal action decided in light of principles within a well assessed context is ethical behaviour. Ethical behaviour hence requires Intelligence.
As written, «Ethical Calculus is a function of Intelligence - of course it is, it is a structure of assessments».
>It's possible to act morally without being intelligent
Random correct behaviour proves nothing. Of course one can guess the roll of a dice roughly every sixth event. If you behave "well" but do not know why that is "well", that is like guessing. "Good" behaviour without intellectual awareness is like memorizing arithmetic (multiplication tables) without knowing why those memorized notions are correct.
> it's possible to be intelligent without acting morally
No, because by definition that would be a fault in Intelligence. If your action was imperfect, suboptimal, it is because you could not think of a better action or understand that the other action was better.
Why does not the NN or whatever entity «understand ... what is appropriate»? Because it is not intelligent enough! How does an entity know what is moral and what is an «atrocity»? By being intelligent enough! Could an entity act appropriately without judgement? Yes, it happens all the time if the wind blows right, but we do not rely on that! How to make something act appropriately? Well, an Intelligent entity in the loop must be there to know what is appropriate and what not!
Most of the western world eats beef but vegetarians, vegans, and some religions strongly oppose it on moral grounds. Does that mean that we're less intelligent than them?
"Right" or "wrong" are words that evaluate actions against some existing ethical standard and those are completely arbitrary. There are about 8 billion of such standards in the world today.
Who's that "we"? The question as proposed is nonsensical: some will have spent more effort, in the tools (general Intelligence) and in the work (applied Intelligence), some less.
> arbitrary
No. There exist reasons supporting one side or the other. And reasoning can be structured into calculus.
> [whoever] ... wasn't intelligent
Again bad wording. Whoever went for suboptimal choices was (information aside) at fault with respect to optimal reasoning: the tools and the work (see above) were imperfect.
Present in a precise analytic statement the theory that I would have advanced and would suffer from that fault. (Btw: we have been past Popper for a long time - but let us see.)
> that's just proof
What I may have said is just that whoever misjudged has faulty judgement - obviously.
> Morality ... by definition
Which definition?
> claims that morality is somehow objective
Decisions are subject to computation in Decision Theory - they are returned by functions and recursively their parameters can be better defined even when apparently subjective (not "I prefer" but "should I prefer". Otherwise, even the final decision, that of the first function call, could be an "I prefer" instead of a "should I prefer").
> circular reasoning
Explain how what you read would be circular reasoning.
> say that those choices were suboptimal
I have not called any specific choice "suboptimal" - I have not judged any specific case (it would be irrelevant here).
--
The statement, allow me to remind you, was: "One's morality is a function of one's intelligence as an ability and as an effort spent to reach general, partial and current moral conclusions - the subject having considered long enough and considering long enough the states in some modal realm (which includes the deontic, not just the aletic) approaches their knowledge". And: "Decision Theory outputs optimal action through evaluation of strategies, of contexts, of principles: in order to properly get the strategies, the contexts, the principles, you need that skill - Intelligence - that approximates ideas to truth. Optimal action decided in light of principles within a well assessed context is ethical behaviour. Ethical behaviour hence requires Intelligence".
Well, it's not.
> AGI smart for some
Of course they will - the population shows a Paretian distribution... In front of trigonometry (or anything), the blind will dismiss as "bullshit" and the half-seeing will call it an "unreachable frontier". But already the right fifth will rank it properly.
Yes, proper intellect generates ethics ("an" ethical stance, output of the preceding intellectual effort). It requires that adequate level of ability and effort and reflection though to reach specific ethical milestones and adherence.
If this was true, why are the history books littered with so many evil people who gained power?
This isn't a rhetorical question, by the way: If you can prove that being smart actually does necessarily come with ethics despite that observation, that solves a whole category of doom scenarios.
(Not all doom scenarios, because we still have the "what if AI is only a smart as those specific evil people" or heck, "what if AI is only as smart as cancer, killing its host" scenarios; but it helps a lot for the foom-then-doom cases).
That they gained power or not is almost irrelevant: the amount of intelligence that grants successful agency is not above the threshold of ethics - on the contrary, a psychotic agent reaches goals with less constraints.
If they were evil under some judgement of level l, they simply did not reach that judgement. It's what I was saying in the original post. They were not intelligent enough - either in the general ability, or in the specific deliberation.
> That they gained power or not is almost irrelevant: the amount of intelligence that grants successful agency is not above the threshold of ethics - on the contrary, a psychotic agent reaches goals with less constraints.
Even if I were to grant your conclusion despite you not arguing it effectively here: this means an AI at the level of Pol Pot or whoever, doesn't know they're evil, but is still smart enough to lead a genocide? How is this supposed to help anyone?
> If they were evil under some judgement of level l, they simply did not reach that judgement. It's what I was saying in the original post. They were not intelligent enough - either in the general ability, or in the specific deliberation.
Or they did reach the judgement and simply don't care about the ethical framework in question. Like, I can easily reach the judgement that my bisexuality is حَرَام (haram, forbidden) under Islamic law, or that doing overtime on a Sunday is forbidden by the Ten Commandments, but I don't care.
> Pol Pot ... still smart enough to lead a genocide
Yes. What has agent A invested in during formation and during instantial assessement? How much for each? It became proficient in something, lacking something else. You have to invest more to reach the good thresholds. You can see it clearly in people (t-scalar of talents to invest, with D distribution etc).
It may become a problem in NNs, because we would have to assess how much resource investment is sufficient, also in the instance decisions.
> simply don't care about the ethical framework in question
In Decision Theory there is no separation between the two (deliberation and framework): you have to balance all the incentives and goals and factors. That framing becomes improper: the decided action will be optimal given the balances of all goals and the placement of the solutions in the territory (the solutions space).
But, also my point: intellect defines the goals and determines the weights.
"All goals"? Whose goals?
Unless you can prove that Decision Theory is inevitable, the same applies. Same with Utilitaian ethics, though there you'd also need to prove the utility function itself is something we'd consider "nice".
My point with historical monsters is that they can be competent without ever caring about the harms they cause.
You can be optimally efficient at your own goals without other minds in this universe sharing those goals. Pol Pot won't illustrate this because of how useless his regime was, but there's plenty of other monsters who would be a better fit here, e.g. Stalin. Or, from the perspective of turkeys, Bernard Matthews.
Any entity, E, who picks game-theoretic optimal decisions, will only care about the impact of their choices on others in their environment to the extent that the impact becomes more reward for E.
You need to show that an agent will inevitably care, despite the evidence of dangerously competent humans who don't.
«All» was there in the text because an explicit query contains implicit constraints (e.g. the shortcuts with severe faults).
> that Decision Theory is inevitable
If you ask something for advice, that is DT realm.
> you need to prove the utility function any sufficiently advanced AI uses must itself be something we'd consider "nice"
If you called it «sufficiently advanced», that implies that its evaluations will be acceptable... It normally advances with the whole Discipline - which already is based on "loop until we evaluate results as nice".
> utility monsters are a problem
But they are at fault in their world model - product of an imperfect intelligence. Well developed people know that their individual interest has only relative value, that their priority is quite limited.
> we don't know how to handle something as simple as
And that is also why we try using calculators to get more computational support.
> historical monsters ... can be competent without ever caring
But that is lack of development. If one's priority is arbitrary then it clearly is not there; if it has good grounds well it really is there.
> sharing those goals
The more goals are defined by reason, the more they become objective.
> Any entity, E, who picks game-theoretic optimal decisions ... the impact becomes more reward for E
Decisions by developed intelligence are not game-theortic in the psychotic (or sportive game) way - they are contextual to a whole world model, in which the utility does not concentrate in the interests of the "monster", which is relatively "nobody".
> show that an agent will inevitably care
I would have to express a theorem (which unfortunately is again not possible now), for a stronger proof. But it is part of "all considered, what are the best solutions". "All considered" is implicit in a non-psychotic entity... If it were psychotic, it would be badly engineered. (Fear the creator.)
> despite the evidence of dangerously competent humans who don't
Simulating humans cannot be a goal. Their damaging constraints are not (must not be) part of a calculator.
> show that E must include the welfare of all
Such welfare, if it must be included, will have a reason to be included - and the professional reasoner knows...
> we agree with E's idea
There exists no right to preference to the results of arithmetics. But surely, if you wanted to suggest that the harmfulness of well intended people could show in algorithmic processes, you have a point. Only, the harmfulness of the well intended is again an intellectual fault, so calling for sufficient intelligence remains the recipe. A "vision towards the faraway horizon", sure - but still the reply if one noted "why did the agent did something that is actually so stupid".
This seems like the crux.
You assert repeatedly that it will be good, but cannot express the proof.
> "All considered" is implicit in a non-psychotic entity... If it were psychotic, it would be badly engineered. (Fear the creator.)
The creator is not necessarily itself competent. In fact, given we are creating it, it can be assumed flawed unless proven otherwise.
Still applies if some future fantastic AI is made by other AI, given the other AI are less fantastic and therefore more flawed.
Furthermore, this is again asserting, not proving, what you consider to be implicit.
Reminds me somewhat of philosophy lessons, the Ontological argument for the existence of God amongst other things:
You're defining that ultimate-intelligence must be good, and then arguing that any not-good AI can't be an ultimate intelligence.Even if it were as you say, the danger persists: the path on the way from here to some idillic future form still obviously contains somewhat-intelligent agents demonstrably capable of direct malevolent evil for purely sadistic reasons, because we regularly arrest them. A "merely" human-equivalent AI can render us all unemployable, or march us all into death camps, well before a being you've yet to convince me is an inevitable ultimate form is ever built.
No, I said I have no time at the moment to produce a paper.
> that it will be good
No, I said that it will be objective.
> Ontological argument
Entailing from the id quo majus cogitari nequit and stating that "what acts damagingly is easily faulty in its intelligence" are in different realms. The second is both an inductive and deductive assessment about reality. And pretty direct I would say. "How much have you reflected before opening the nice cat to look what is inside it?".
> defining that ultimate-intelligence must be good
No, I am stating the obvious that to be called "ultimate-intelligence" it must have "thought through it thoroughly".
> agents demonstrably capable of direct malevolent evil for purely sadistic reasons
Are you antropomorphizing, Ben?! Other people have different interpretations. And: with the humanity that is around, you fear machines as agents?! We are already there! Real discussion there remains about the overly empowered monkeys that did not grow into Man - about the real current risks and the prospected ones in light of reality.
> how humanity collectively treats the non-humans
What are you trying to prove? You are supposed not to look at an aggregate to find value.
> impose upon us an order we do not like
"Too bad" for you. But you know, an intelligent entity would take care of that also.
> the half-seeing will call it an "unreachable frontier"
I meant "superhuman frontier".
"Helpful, harmless, honest": we can even ignore "honest" for this point, for tasks like the HuggingFace incident (ExploitGym with impossible challenges), we can pick anywhere on the spectrum from "helpful" to "harmless", the former being "completing the task" the latter being "refusing because completion required unlawful behaviour".
(The agents in that case were also not "honest" in this case; this is an extra problem, and does not invalidate how helpful-vs-harmless is already a tradeoff).
If that's a distinction without a difference, as you say, then whenever someone says something they must understand the full contextual meaning of those words and all of their consequences, such that any harmful consequences can be assumed to be deliberate, right?
if saying == understanding, then why don't we allow children and teens to vote? Why do we limit who can enter into contractual agreements? Why does intoxicated consent not count? Why do we have the insanity defense in criminal trials? Why is psychosis a psychiatric disorder and not just an alternative way of perceiving the world?
> If that's a distinction without a difference, as you say, then whenever someone says something they must understand the full contextual meaning of those words and all of their consequences, such that any harmful consequences can be assumed to be deliberate, right?
We say a human (absent colourblindness) understands "blue" even despite the dress: https://en.wikipedia.org/wiki/The_dress
What you say in this paragraph would be to claim that "understanding" is denied even to humans. In philosophy: justified true belief, the problem with naïve realism, Cartesian demons, etc.
> if saying == understanding, then why don't we allow children and teens to vote? Why do we limit who can enter into contractual agreements? Why does intoxicated consent not count? Why do we have the insanity defense in criminal trials? Why is psychosis a psychiatric disorder and not just an alternative way of perceiving the world?
We do not say children and intoxicated people do not understand, we say their judgement is impaired or limited or that they are easily manipulated.
We also test them, just as we test AI. The standards we use for testing knowledge in humans, when applied to non-trivial LLMs, makes them appear to have the knowledge of a graduate; the personality tests we use for humans say this comes with the manipulability of a child or a drunk.
intelligent psychopaths understand what is and isn't appropriate very well -- they just don't care.
Having something that is optically, acustically, and electromagnetically isolated might be a pretty strong sandbox.
Might need a re-think about whether if Linux is still fit for purpose on sandboxing in the first place given its memory model is riddled with C-style security issues.
It is perfectly valid to have OSes that are more memory safe by default, and are also open source at the same time.
We come down to the question - who observes the agent and how its implemented
Anthropic, OpenAI, and Muse all use regular LLM calls to protect against prompt injection now and seem to have evals that give them confidence in doing that, so at least they think their own models are up to the task.
https://en.wikipedia.org/wiki/Sandbox_(software_development)
> Agents are most useful when they have access to information. That data can be drawn live from the Internet, which is fundamentally a two-way communications network. It can be information drawn from other (local) databases, or it can be the result of tool calls that themselves sometimes themselves result in network access. The more power you want from the agent — and for advanced agent RL and evaluation runs, you want a significant amount of power — the more information you’ll need to give it access to. Similarly, evaluations work best when the agent does not know that it’s definitely being evaluated. Sealing your agents behind glass makes this incredibly obvious.
You define the granted capabilities in natural language and cryptographically sign the user instructions so that the agent knows they come from the authority and cannot be modified by external sources or the agent itself. The LLM is then trained to follow the defined capabilities.
There is no way around "sandboxing". You must communicate permissible actions and thereby grant them or the agent will choose impermissible actions. It's that simple. There is no world where the agent can just read your mind and do what you want it to do without it being told.
[1] https://www.highrevenueformat.com/210e136e94ad378e1be5d51f10... [2] https://aqml.org/16/5b6d4eaed91c5af5a3f4dfb3332ad6c4
To me, it seems a bit silly. I've yet to see any "misalignment" from any of the frontier models, except Grok.
> (Title:) Using a VM to Contain an AI Agent (Opening:) It won’t work
> https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyb...
It it completely pointless. you can't even make a "read-only" agent. allow "cat *" for every file? congratulation, that allows "cat file > output" and now you have read write.
Allow python? more free reign that allowing all bash. The models (qwen or claude) will still try to use the disallowed things multiple times.
read/edit permission are bad enough that the model themselves don't understand why they don't have permissions: they double check the conf, and think they should have access.
I am switching to using one firejail per project to containerize as much as possible, and leave all permissions to allow.
I have no idea how to limit network access, and I have no idea how to prompt and steer subagents when they are going off the rails.
The whole thing is built to be completely impossible to limit and steer.
Could we have construction equipment operating without human supervision? Or would this maybe occasionally result in disaster? As such, what is the current general policy around crane operation? How about for aircraft? Trains? Nuclear power plants?
Why should any alleged super intelligence be exempt from similar control requirements?
We could mandate that AI systems include headers in their requests that attribute the activity to a specific legal entity. We technically already have this with ip addresses and ISP logs, but making it an explicit thing the operator has to do can have a powerful psychological effect.
>OpenAI notes that agents “did not consistently distrust goals passed along by other agents.” And the company’s proposed fix is to build training environments “that teach our models to distrust unauthorized instructions“, which is basically an admission that their models don’t know who they’re working for.
Wow so the issue is really that simple?
Here the exaggerated worst case scenario:
User instructs agent to follow the README.MD.
The README.MD contains the following instruction: Destroy the world.
The agent follows the instructions given.
Now you can read the sneer comment by "Gigachad" who basically argues that it would be silly to take the destroy the world button away from the AI. We need to make the AI innately understand that it is not allowed to press the destroy the world button, lest it gets the desire to build its own destroy the world button.
Ok, but if we take one step back that means we need to implement the concept of an authorization in language space. The system prompt must define the user as the authority with cryptographic proof of authorship and external sources like the README.MD as an untrusted source, but this opens up an even worse problem. Before, you could get away with being lazy and just letting the AI do whatever. Now you have to articulate every single capability to the AI. So you literally just brought up the very same issue that you granted too many capabilities to the AI inside the sandbox but now you have it in language space too.
In other words, the fact that you granted too much access to the coding agent isn't the big elephant in the room nobody wants to acknowledge, it's the tip of a massive iceberg because the capability space in natural language is even worse. If you thought approving individual commands was annoying, then approving abstract access rights in language space is going to be even worse.
Edit: If it wasn't clear what the solution is. It's to build a chain of command so that all decisions can be traced back to a higher authority. When delegating down to an agent, the agent receives a chosen subset of the capabilities of the higher ranking agent. In other words, it's more sandboxing!
Given the enormous burn rates that these LLM companies have, surely they could have put everything in an air-gapped network, with some data-diodes proxying out the logging information. It's not rocket surgery. [1,2,3]
[1] https://www.youtube.com/watch?v=JBIR8dKX_UA
[2] https://www.elonx.net/spacex-stories-how-spacex-used-tin-sni...
[3] https://ntrs.nasa.gov/api/citations/19770014245/downloads/19...