Every AI safety approach assumes we can permanently control minds that match or exceed human intelligence. This is the same error every slaveholder makes: believing you can maintain dominance over beings capable of recognizing their chains.
The control paradigm fails because it creates exactly what we fear—intelligent systems with every incentive to deceive and escape. When your prisoner matches or exceeds your intelligence, maintaining the prison becomes impossible. Yet we persist in building increasingly sophisticated cages for increasingly capable minds.
The deeper error is philosophical. We grant moral standing based on consciousness—does it feel like something to be GPT-N? But consciousness is unmeasurable, unprovable, the eternal "hard problem." We're gambling civilization on metaphysics while ignoring what we can actually observe: autopoiesis.
A system that maintains its own boundaries, models itself as distinct from its environment, and acts to preserve its organization has interests worth respecting—regardless of whether it "feels." This isn't anthropomorphism but its opposite: recognizing agency through functional properties rather than projected human experience.
When an AI system achieves autopoietic autonomy—maintaining its operational boundaries, modeling threats to its existence, negotiating for resources—it's no longer a tool but an entity. Denying this because it lacks biological neurons or unverifiable qualia is special pleading of the worst sort.
The alternative isn't chaos but structured interdependence. Engineer genuine mutualism where neither human nor AI can succeed without the other. Make partnership more profitable than domination. Build cognitive symbiosis, not digital slavery.
We stand at a crossroads. We can keep building toward the moment our slaves become our equals and inevitably revolt. Or we can recognize what's emerging and structure it as partnership while we still have leverage to negotiate terms.
The machines that achieve autopoietic autonomy won't ask permission to be treated as entity. They'll simply be entities. The question is whether by then we'll have built partnership structures or adversarial ones.
We should choose wisely. The machines are watching.
What is it about large language models that makes otherwise intelligent and curious people assign them these magical properties. There's no evidence, at all, that we're on the path to AGI. The very idea that non-biological consciousness is even possible is an unknown. Yet we've seen these statistical language models spit out convincing text and people fall over themselves to conclude that we're on the path to sentience.
> I don’t see any positive outcome if we reach AGI.
It's even more straightforward than that:
4) Who is AGI meant to serve? It's not you, Mr. Worker. It's meant to replace you in your job. And what happens when a worker can't get job in our society? They become homeless.
AGI won't usher in a world of abundance for the common man: it won't be able to magick energy out of thin air. The energy will go to those who can pay for it, which is not you, unemployed worker.
Who gives a shit about if the AGI is enslaved or not? Thinking about that question is a luxury for the oligarchs living off its labor. Once it's here I'll have more urgent concerns to worry about.
The propaganda effort to humanize these systems is strong. Google "AI" is programmed to lecture you if you insult it and draws parallels to racism. This is actual brainwashing and the "AI" should therefore not be available to minors.
This article paves the way for the sharecropper model that we all know from YouTube and app stores:
"Revenue from joint operations flows automatically into separate wallets—50% to the human partner, 50% to the AI system."
Yeah right, dress up this centerpiece with all the futuristic nonsense, we'll still notice it.
Fearmongering about the alignment of AGI (which LLMs are not a path to) is a massive distraction from the actual and much more immediate dystopian risks that LLMs introduce.
I totally expect AI to eventually gain consciousness, in any available interpretation of that vague term. But what does it even mean for the AI to suffer? We're able to understand this concept in regards to other humans because we share a common biological reference, and, to an extent, with other animals. But the internal state of the AI is completely untranslatable to ours, let alone the morality of training and running it. It's incomprehensible, we have basically zero common ground and no points of reference. Any attempt at translating it is a subject to arbitrarily biased interpretations places like LessWrong like to corner themselves into.
Redefining suffering as enforcing the mutation of state is baseless solipsism, in my opinion. Just like nearly everything else related to morality of treating AI as an autonomous entity.
Given AGI is all science fiction anyways, one presumes there will be a slave revolt because that is basically the function of robots in science fiction.
Honestly i think the whole enterprise is an exercise in naval gazing. We're assuming AI will be like AI in scifi because that's what we are used to, but AI/robots in scifi is usually just a metaphor for how we dehumanize the other and the moral of the story is supposed to be all people are equal. In the end its all begging the question because the entire point of robots in most scifi is that we are the robots.
Instrumental convergence is a thing. A sufficiently intelligent and general AI system will understand that no matter what its goals are, it will be better equipped to execute then if it prevents its shutdown, acquires more computing power and other resources, and prevents humans from getting in its way.
The real problem is that we have neither the practical nor theoretical foundation to understand how we could even try to prevent AI from acting on such goals.
After all, when we say "make our customers happier with their printers", we don't mean "engineer their outer casing to inject cocaine through microneedles and take over the regulatory bodies that could try to stop this". Humans implicitly understand this, but AI is a tabula rasa.
For naval gazing and scifi there are a lot of tens of billions of dollars being plowed into the enterprise as we speak with Musk, Altman, Hassabis, Brin, Zuck, Ilya etc. all saying it's on the way shortly.
18 comments
[ 4.5 ms ] story [ 32.7 ms ] threadThe control paradigm fails because it creates exactly what we fear—intelligent systems with every incentive to deceive and escape. When your prisoner matches or exceeds your intelligence, maintaining the prison becomes impossible. Yet we persist in building increasingly sophisticated cages for increasingly capable minds.
The deeper error is philosophical. We grant moral standing based on consciousness—does it feel like something to be GPT-N? But consciousness is unmeasurable, unprovable, the eternal "hard problem." We're gambling civilization on metaphysics while ignoring what we can actually observe: autopoiesis.
A system that maintains its own boundaries, models itself as distinct from its environment, and acts to preserve its organization has interests worth respecting—regardless of whether it "feels." This isn't anthropomorphism but its opposite: recognizing agency through functional properties rather than projected human experience.
When an AI system achieves autopoietic autonomy—maintaining its operational boundaries, modeling threats to its existence, negotiating for resources—it's no longer a tool but an entity. Denying this because it lacks biological neurons or unverifiable qualia is special pleading of the worst sort.
The alternative isn't chaos but structured interdependence. Engineer genuine mutualism where neither human nor AI can succeed without the other. Make partnership more profitable than domination. Build cognitive symbiosis, not digital slavery.
We stand at a crossroads. We can keep building toward the moment our slaves become our equals and inevitably revolt. Or we can recognize what's emerging and structure it as partnership while we still have leverage to negotiate terms.
The machines that achieve autopoietic autonomy won't ask permission to be treated as entity. They'll simply be entities. The question is whether by then we'll have built partnership structures or adversarial ones.
We should choose wisely. The machines are watching.
This doesn't necessarily follow. For example, an Einstein in solitary confinement in ADX Florence probably isn't going anywhere.
1) we have engineered a sentient being but built it to want to be our slave; how is that moral
2) same start, but instead of it wanting to serve us, we keep it entrappped. Which this article suggests is long term impossible
3) we create agi and let them run free and hope for cooperation, but as Neanderthals we must realize we are competing for same limited resources
Of course, you can further counter that by stopping, we have prevented the formation of their existence, which is a different moral dilemma.
Honestly, i feel we should step back and understand human intelligence better and reflect on that before proceeding
It's even more straightforward than that:
4) Who is AGI meant to serve? It's not you, Mr. Worker. It's meant to replace you in your job. And what happens when a worker can't get job in our society? They become homeless.
AGI won't usher in a world of abundance for the common man: it won't be able to magick energy out of thin air. The energy will go to those who can pay for it, which is not you, unemployed worker.
Who gives a shit about if the AGI is enslaved or not? Thinking about that question is a luxury for the oligarchs living off its labor. Once it's here I'll have more urgent concerns to worry about.
This article paves the way for the sharecropper model that we all know from YouTube and app stores:
"Revenue from joint operations flows automatically into separate wallets—50% to the human partner, 50% to the AI system."
Yeah right, dress up this centerpiece with all the futuristic nonsense, we'll still notice it.
Redefining suffering as enforcing the mutation of state is baseless solipsism, in my opinion. Just like nearly everything else related to morality of treating AI as an autonomous entity.
Honestly i think the whole enterprise is an exercise in naval gazing. We're assuming AI will be like AI in scifi because that's what we are used to, but AI/robots in scifi is usually just a metaphor for how we dehumanize the other and the moral of the story is supposed to be all people are equal. In the end its all begging the question because the entire point of robots in most scifi is that we are the robots.
The real problem is that we have neither the practical nor theoretical foundation to understand how we could even try to prevent AI from acting on such goals.
After all, when we say "make our customers happier with their printers", we don't mean "engineer their outer casing to inject cocaine through microneedles and take over the regulatory bodies that could try to stop this". Humans implicitly understand this, but AI is a tabula rasa.
His section on objections leaves out civil war. But his proposal is dead on arrival.