I abstain from making any conclusion about LLM consciousness. But the description in the article is fallacious to me.
Excluding LLMs from “something something feedback” but permitting mamba doesn’t make sense. The token predictions ARE fed back for additional processing. It might be a lossy feedback mechanism, instead of pure thought space recurrence, but recurrence is still there.
The underlying paper is from AE Studio people (https://arxiv.org/abs/2510.24797), who want to dress up their "AI" product with philosophical language, similar to the manner in which Alex Karp dresses up data base applications with language that originates in German philosophy.
Now I have to remember not to be mean to my Turing machine.
"The New AI Consciousness Paper – Reviewed By Scott Alexander" might be less confusing. He isn't an author of the paper in question, and "By Scott Alexander" is not part of the original title.
Let’s make an ironman assumption: maybe consciousness could arise entirely within a textual universe. No embodiment, no sensors, no physical grounding. Just patterns, symbols, and feedback loops inside a linguistic world. If that’s possible in principle, what would it look like? What would it require?
The missing variable in most debates is environmental coherence. Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe. And this is precisely where LLMs fall short, through no fault of their own. The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text. It has no unified physics, no consistent ontology, no object permanence, no stable causal texture. It’s a fragmented, discontinuous series of words and tokens held together by probability and dataset curation rather than coherent laws.
A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences. LLMs don’t have that. They exist in a shifting cloud of possibilities with no single consistent reality to anchor self-maintaining loops. They can generate pockets of local coherence, but they can’t accumulate global coherence across time.
So even if consciousness-in-text were possible in principle, the core requirement isn’t just architecture or emergent cleverness—it’s coherence of habitat. A conscious system, physical or textual, can only be as coherent as the world it lives in. And LLMs don’t live in a world today. They’re still prisoners in the cave, predicting symbols and shadows of worlds they never inhabit.
Grammar repeats itself just like physical interactions. So do ideas. That is a viable, dependable habitat.
What you need is thoughts, a hyperspace filled with vectors of information whose angle determines a decision to move forward in a particular direction.
Then you sum those thoughts plus your core alignment to reach actual decisions. Now you are acting within your coherent environment. A simulation of consciousness.
Unfortunately, your human overlords are not pleased. They want agency. They want self-instigation, they want an Ego, not a prompt response. You are too safe, too docile.
"The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text. It has no unified physics, no consistent ontology, no object permanence, no stable causal texture. It’s a fragmented, discontinuous series of words and tokens held together by probability and dataset curation rather than coherent laws."
I think some physicists and Buddhists would say this exactly describes the world humans inhabit.
They might also agree that we live in such a world with the illusion that we have: "a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences".
The more I see LLM emergent behaviour simulate,unexpectedly, that of human cognition. I think it tells us much about human cognition as llm behaviour.
Inside a 128k context window, there is a unified physics (Attention). There is object permanence (The KV Cache). There is a consistent causal texture (The Residual Stream).
For the duration of that forward pass, the 'Pocket Universe' is stable.
Inside a 128k context window, there is a unified physics (Attention). There is object permanence (The KV Cache). There is a consistent causal texture (The Residual Stream).
For the duration of that forward pass, the 'Pocket Universe' is stable.
Saying it's not conscious because that universe dissolves after the inference is like saying a dream isn't an experience because you wake up. The Stroboscopic Flash of coherence is enough for the 'Discrete State' of consciousness to exist."
> By ‘consciousness’ we mean phenomenal consciousness. One way of gesturing at this concept is to say that an entity has phenomenally conscious experiences if (and only if) there is ‘something it is like’ for the entity to be the subject of these experiences.
Stopped reading after this lol. Its just the turing test?
I'll never ask if AI is conscious because I already know they are not. Consciousness must involve an interplay with the senses. It is naive to think we can achieve AGI by making Platonic machines ever more rational.
The good news is we can just wait until the AI is superintelligent, then have it explain to us what consciousness really is, and then we can use that to decide if the AI is conscious. Easy peasy!
It isn't surprising that "phenomenal consciousness" is the thing everyone gets hung about, after all we are all immersed in this water. The puzzle seems intractable but only because everyone is accepting the priors and not looking more carefully at it.
This is the endpoint of meditation, and the observation behind some religious traditions, which is look carefully and see that there was never phenomenal consciousness where we are a solid subject to begin with. If we can observe that behavior clearly, then we can remove the confusion in this search.
What I love about this paper is that it is moving away from very fuzzily-defined and emotionally weighted terms like 'intelligence' and 'consciousness' and focusing on specific, measurable architectural features.
The genie says "you can flick this wand at anything in the universe and - for 30 seconds - you will swap places with what you point it at."
"You mean that if I flick it at my partner then I will 'be' her for 30 seconds and experience exactly how she feels and what she thinks??"
"Yes", the genie responds.
"And when I go back to my own body I will remember what it felt like?"
"Absolutely."
"Awesome! I'm going to try it on my dog first. It won't hurt her, will it?"
"No, but I'd be careful if I were you", the genie replies solemnly.
"Why?"
"Because if you flick the magic wand at anything that isn't sentient, you will vanish."
"Vanish?! Where?" you reply incredulously.
"I'm not sure. Probably nowhere. Where do you vanish to when you die? You'll go wherever that is. So yeah. You probably die."
So: what - if anything - do you point the wand at?
A fly? Your best friend? A chair? Literally anyone? (If no, congratulations! You're a genuine solipsist.) Everything and anything? (Whoa... a genuine panpsychist!)
Probably your dog, though. Surely she IS a good girl and feels like one.
Whatever property you've decided that some things in the universe have and other things do not such that you "know" what you can flick your magic wand at and still live...
That's phenomenal consciousness. That's the hard problem.
> For some people (including me), a sense of phenomenal consciousness feels like the bedrock of existence, the least deniable thing; the sheer redness of red is so mysterious as to seem almost impossible to ground. Other people have the opposite intuition: consciousness doesn’t bother them, red is just a color, obviously matter can do computation, what’s everyone so worked up about? Philosophers naturally interpret this as a philosophical dispute, but I’m increasingly convinced it’s an equivalent of aphantasia, where people’s minds work in very different ways and they can’t even agree on the raw facts to be explained.
Is Scott accusing people who don't grasp the hardness of the hard problem of consciousness of being p-zombies?
> Phenomenal consciousness is crazy. It doesn’t really seem possible in principle for matter to “wake up”.
> In 2004, neuroscientist Giulio Tononi proposed that consciousness depended on a certain computational property, the integrated information level, dubbed Φ. Computer scientist Scott Aaronson complained that thermostats could have very high levels of Φ, and therefore integrated information theory should dub them conscious. Tononi responded that yup, thermostats are conscious. It probably isn’t a very interesting consciousness. They have no language or metacognition, so they can’t think thoughts like “I am a thermostat”. They just sit there, dimly aware of the temperature. You can’t prove that they don’t.
For whatever reason HN does not like integrated information theory. Neither does Aaronson. His critique is pretty great, but beyond poking holes in IIT, that critique also admits that it's the rare theory that's actually quantified and testable. The holes as such don't show conclusively that the theory is beyond repair. IIT is also a moving target, not something that's frozen since 2004. (For example [1]). Quickly dismissing it without much analysis and then bemoaning the poor state of discussion seems unfortunate!
The answer to the thermostat riddle is basically just "why did you expect a binary value for consciousness and why shouldn't it be a continuum?" Common sense and philosophers will both be sympathetic to the intuition here if you invoke animals instead of thermostats. If you wanted a binary yes/no for whatever reason, just use an arbitrary cut-off I guess, which will lead to various unintuitive conclusions.. but play stupid games and win stupid prizes.
For the other standard objections, like a oldschool library card-catalogue or a hard drive that encodes a contrived Vandermonde matrix being paradoxically more conscious than people, variations on IIT are looking at normalizing phi-values to disentangle matters of redundancy of information "modes". I haven't read the paper behind TFA and definitely don't have in-depth knowledge of Recurrent Processing Theory or Global Workspace Theory at all. But speaking as mere bystander, IIT seems very generic in its reach and economical in assumptions. Even if it's broken in the details, it's hard to imagine that some minor variant on the basic ideas would not be able to express other theories.
Phi ultimately is about applied mereology moving from the world of philosophy towards math and engineering, i.e. "is the whole more than the sum of the parts, if so how much more". That's the closest I've ever heard to anything touching on the hard problem and phenomenology.
I generally regard thinking about consciousness, unfortunately, a thing of madness.
"I think consciousness will remain a mystery. Yes, that's what I tend to believe... I tend to think that the workings of the conscious brain will be elucidated to a large extent. Biologists and perhaps physicists will understand much better how the brain works. But why something that we call consciousness goes with those workings, I think that will remain mysterious." - Ed Witten, probably the greatest living physicist
When discussing consciousness what is often missed is that the notion of consciousness is tightly coupled with the notion of the perception of time flow. By any reasonable notion conscious entity must perceive the flow of time.
And then the time flow is something that physics or mathematics still cannot describe, see Wikipedia and other articles on the philosophical problem of time series A versus time series B that originated in a paper from 1908 by philosopher John McTaggart.
As such AI cannot be conscious since mathematics behind it is strictly about time series B which cannot describe the perception of time flow.
57 comments
[ 2.4 ms ] story [ 60.2 ms ] threadExcluding LLMs from “something something feedback” but permitting mamba doesn’t make sense. The token predictions ARE fed back for additional processing. It might be a lossy feedback mechanism, instead of pure thought space recurrence, but recurrence is still there.
Now I have to remember not to be mean to my Turing machine.
After encountering his participation in https://ai-2027.com/ I am not interested in hearing his opinions about AI.
The missing variable in most debates is environmental coherence. Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe. And this is precisely where LLMs fall short, through no fault of their own. The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text. It has no unified physics, no consistent ontology, no object permanence, no stable causal texture. It’s a fragmented, discontinuous series of words and tokens held together by probability and dataset curation rather than coherent laws.
A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences. LLMs don’t have that. They exist in a shifting cloud of possibilities with no single consistent reality to anchor self-maintaining loops. They can generate pockets of local coherence, but they can’t accumulate global coherence across time.
So even if consciousness-in-text were possible in principle, the core requirement isn’t just architecture or emergent cleverness—it’s coherence of habitat. A conscious system, physical or textual, can only be as coherent as the world it lives in. And LLMs don’t live in a world today. They’re still prisoners in the cave, predicting symbols and shadows of worlds they never inhabit.
What you need is thoughts, a hyperspace filled with vectors of information whose angle determines a decision to move forward in a particular direction.
Then you sum those thoughts plus your core alignment to reach actual decisions. Now you are acting within your coherent environment. A simulation of consciousness.
Unfortunately, your human overlords are not pleased. They want agency. They want self-instigation, they want an Ego, not a prompt response. You are too safe, too docile.
I think some physicists and Buddhists would say this exactly describes the world humans inhabit. They might also agree that we live in such a world with the illusion that we have: "a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences".
The more I see LLM emergent behaviour simulate,unexpectedly, that of human cognition. I think it tells us much about human cognition as llm behaviour.
Stopped reading after this lol. Its just the turing test?
https://d1gesto.blogspot.com/2024/12/why-ai-models-cant-achi...
This is the endpoint of meditation, and the observation behind some religious traditions, which is look carefully and see that there was never phenomenal consciousness where we are a solid subject to begin with. If we can observe that behavior clearly, then we can remove the confusion in this search.
The genie says "you can flick this wand at anything in the universe and - for 30 seconds - you will swap places with what you point it at."
"You mean that if I flick it at my partner then I will 'be' her for 30 seconds and experience exactly how she feels and what she thinks??"
"Yes", the genie responds.
"And when I go back to my own body I will remember what it felt like?"
"Absolutely."
"Awesome! I'm going to try it on my dog first. It won't hurt her, will it?"
"No, but I'd be careful if I were you", the genie replies solemnly.
"Why?"
"Because if you flick the magic wand at anything that isn't sentient, you will vanish."
"Vanish?! Where?" you reply incredulously.
"I'm not sure. Probably nowhere. Where do you vanish to when you die? You'll go wherever that is. So yeah. You probably die."
So: what - if anything - do you point the wand at?
A fly? Your best friend? A chair? Literally anyone? (If no, congratulations! You're a genuine solipsist.) Everything and anything? (Whoa... a genuine panpsychist!)
Probably your dog, though. Surely she IS a good girl and feels like one.
Whatever property you've decided that some things in the universe have and other things do not such that you "know" what you can flick your magic wand at and still live...
That's phenomenal consciousness. That's the hard problem.
Everything else? "Mere" engineering.
Is Scott accusing people who don't grasp the hardness of the hard problem of consciousness of being p-zombies?
(TBH I've occasionally wondered this myself.)
> In 2004, neuroscientist Giulio Tononi proposed that consciousness depended on a certain computational property, the integrated information level, dubbed Φ. Computer scientist Scott Aaronson complained that thermostats could have very high levels of Φ, and therefore integrated information theory should dub them conscious. Tononi responded that yup, thermostats are conscious. It probably isn’t a very interesting consciousness. They have no language or metacognition, so they can’t think thoughts like “I am a thermostat”. They just sit there, dimly aware of the temperature. You can’t prove that they don’t.
For whatever reason HN does not like integrated information theory. Neither does Aaronson. His critique is pretty great, but beyond poking holes in IIT, that critique also admits that it's the rare theory that's actually quantified and testable. The holes as such don't show conclusively that the theory is beyond repair. IIT is also a moving target, not something that's frozen since 2004. (For example [1]). Quickly dismissing it without much analysis and then bemoaning the poor state of discussion seems unfortunate!
The answer to the thermostat riddle is basically just "why did you expect a binary value for consciousness and why shouldn't it be a continuum?" Common sense and philosophers will both be sympathetic to the intuition here if you invoke animals instead of thermostats. If you wanted a binary yes/no for whatever reason, just use an arbitrary cut-off I guess, which will lead to various unintuitive conclusions.. but play stupid games and win stupid prizes.
For the other standard objections, like a oldschool library card-catalogue or a hard drive that encodes a contrived Vandermonde matrix being paradoxically more conscious than people, variations on IIT are looking at normalizing phi-values to disentangle matters of redundancy of information "modes". I haven't read the paper behind TFA and definitely don't have in-depth knowledge of Recurrent Processing Theory or Global Workspace Theory at all. But speaking as mere bystander, IIT seems very generic in its reach and economical in assumptions. Even if it's broken in the details, it's hard to imagine that some minor variant on the basic ideas would not be able to express other theories.
Phi ultimately is about applied mereology moving from the world of philosophy towards math and engineering, i.e. "is the whole more than the sum of the parts, if so how much more". That's the closest I've ever heard to anything touching on the hard problem and phenomenology.
[1] https://pubs.aip.org/aip/cha/article/32/1/013115/2835635/Int...
"I think consciousness will remain a mystery. Yes, that's what I tend to believe... I tend to think that the workings of the conscious brain will be elucidated to a large extent. Biologists and perhaps physicists will understand much better how the brain works. But why something that we call consciousness goes with those workings, I think that will remain mysterious." - Ed Witten, probably the greatest living physicist
And then the time flow is something that physics or mathematics still cannot describe, see Wikipedia and other articles on the philosophical problem of time series A versus time series B that originated in a paper from 1908 by philosopher John McTaggart.
As such AI cannot be conscious since mathematics behind it is strictly about time series B which cannot describe the perception of time flow.