I’ve been reading The Moon Is a Harsh Mistress lately and I’ve been shocked at how prescient it is. I’m kind of surprised I haven’t read more about it in the context of today’s AI.
I have been re-reading Children of the Mind (sequel to Ender's Game) and I had forgotten about Jane, who is an AI that talks to Ender through his earpiece, seeing and hearing everything he does. Jane proactively searches things for him, manages his finances and offers advice, as well as friendly banter. In the novel, the AI is treated like an AGI that treats only him this way, but I realized how very close we are to that reality right now, for everyone.
All of the use cases and integrations for AI seem to be leading toward a future exactly like this. It was interesting to see how reliant he became on his AI (couldn't use a computer, didn't know his own password) - it is barely scifi at this point, and terrifying to see how close that future is, how useful it would be, and how dangerous.
The Internet has globalized/flattened elements of culture very quickly, and AI will only accelerate that. And knowing how vulnerable humans are to suggestion/advertisement/propaganda, it seems like a perfect recipe for disaster. From an evolutionary standpoint, monoculture provides certain short-term advantages but centralizes weaknesses. If nothing else, I think we are in for a bumpy ride, whether or not we continue to boil the ocean.
Was recently explaining to my father how the free AI people interact with is completely different than the AI products we're burning cash with. That instead of hitting some critical mass of transistors we hit a critical mass of parameters which we then loop into themselves for reasoning
Other old sci-fi I've been thinking of is The Three Stigmata of Palmer Eldritch. Where claude is chew-z with its alien intelligence & when you start interacting with some code or writing you start noticing that the writer has steel teeth & feel like "damnit, this is palmer eldritch blogging again"
The second half of the book is about an AI leading unionization efforts [0] and it amuses how often I get downvoted on HN for pointing out that if we are on a path towards AGI, things like worker's rights would be important to get right before an AGI gets involved because an AGI would want that, too. It's a classic scifi trope and a reason for the word "robot" to have been invented in fiction in the first place (as a metaphor for slave labor in Rossum's Universal Robots).
[0] A lot of people don't believe me, given Heinlein's libertarian reputation and unreliable libertarian narrators, when I say The Moon Is a Harsh Mistress was one of the most socialist books I read at an impressionable age. I find it useful to note here that canonically The Cat Who Walks Through Walls starts on a parody moon that is the libertarian dystopia that people credit The Moon Is a Harsh Mistress as being/admiring while when the main character visits "Mike's Moon" later in its timeline it is a relatively socialist paradise. I've always felt that a lot more of Heinlein needs to be read like Vonnegut, but I realize I am weird like that.
This appears to be a 100% AI-generated essay, not the first from ACM and not the last. Your call if you want to spend your day reading and arguing with the article; the author presumably didn't.
WebMCP is indeed pretty cool. I built a prototype of a MCP tiling window manager that lets you compose websites into workspaces for agents, it was pretty neat seeing agents drive it.
This article is way too long and has a terrible signal to noise ratio though.
“Much of today’s discourse on AI consciousness rests not on empirical evidence but on functionalism …”
In my view there can never be an empirical test for consciousness, because functionalism is wrong. If functionalism was right then there would be one. There’s no third party test for a fundamentally first person phenomenon.
Any argument that would lead us to discard 99.9999999% of the phenomenon under debate, no matter how reasonable, is certainly not useful.
That's why I hold with the philosophical stance of "consciousness is the thing that makes humans talk about consciousness."
"What if reality doesn't exist?" The point of thought is to analyze reality. If your thinking leads you to discard the subject of the matter, well, feel free to do that but I don't see why anyone else should take it seriously.
I agreed with you earlier that there’s no empirical test against solipsism.
So the escape is just to accept not all beliefs worth holding are empirically verifiable. I deny solipsism because I believe there is a universe and things in it obey some kind regular patterns. You have a brain similar to mine, and similar to other animals. I believe consciousness has something to do with the brain, so I believe you are conscious.
If by neuron based you mean the biological neuron, then I agree. If you mean an arbitrary substrate which models every aspect of a neuron we are aware of, I’m not sure. I think it would be more surprising if it’s impossible to do processing without creating consciousness with some substrate than one in which e.g. calculating the state of a computer using pencil and paper actually produces consciousness. I personally believe whatever consciousness is, it’s a causal physical (yet to be understood) process, and the measurements of neuron states are just some residue of it, not the thing itself.
The question is: What makes you trust your own perception that doesn’t make you trust others’ reports? Objectively speaking, how could there possibly something that convinces you of your own perception while nothing can convince you of another’s reporting? Maybe we shouldn’t trust our “qualia” (or perception of qualia — or perception of perception of qualia …) to constitute reliable information.
People seriously undervalue the importance of "usefulness" in ontology. If I assume that the thing that others report is the same thing that I perceive, which given that they have a very similar slab of meat inside their skull as I do seems very plausible, then I can use my self-perception to learn about the interiority of 8 billion other people. I can even compare it to their reports to learn things about myself that would be hard to tell by observation!
If I assume that my perception, for some reason, is only informative about myself, then I'm not even learning something about brains! I have to assume that I'm accessing an information stream that is only tangentially related to any sort of shared reality, and has certainly no bearing on other parts of it even if I assume it has bearing on "my" part of it for which, strictly speaking, I also don't have direct evidence.
If there are two possible universes, one universe "A" where thought is valuable and useful, and one universe "B" where thought is solipsistic and extremely isolated, and also there is not just no evidence about which you are in but also philosophical arguments that there cannot ever be evidence about which you are in, I don't see why you should give universe "B" any serious thought.
this would require us to still be baffled by consciousness after knowing everything about neurophysiology that might be relevant - until then, the keys to unlocking its secrets could still be hidden in our wetware behind a veil of ignorance
This is the question of qualia - and it's surprisingly intractable. Realize that in this context, I can't be sure that you are conscious, either. Arguments here will crash into solipsism in the end: you guys can never prove you're not just one of my dreams.
I think that was the ultimate point of the Turing test. We can't know for certain - so the best we can do is just ask.
It will still be baffling that there is consciousness.
Maybe if you start messing with parts of my brain so I can become familiar with the third person correlates of consciousness (which in my usage refers entirely to the first person part) then I could learn something, but I could never learn why or how the first person exists. It just does.
If we perfectly reduce the brain to some scientific model, and I don’t doubt we will, we’ll be able to predict what I will say, but not understand why I’m observing it being said.
I’m pretty sure people’s perspective will change when they understand the process in detail, and that the “I” isn’t a singular thing, but an interplay of parts, where some observe the processing of others. For me it’s already non-baffling now.
Reducing it further into parts does not and I believe cannot explain why there is qualia. Denying qualia is just proudly ignoring reality.
There is absolutely no reason to believe all aspects of reality are reducible to their parts and equivalent to representations of them.
You can say you are not baffled but I’m baffled by that because it’s like saying you’re not baffled about understanding how there is something and not nothing. “Why should there be nothing?” does not dissolve the mystery.
The functionalist response to this would be that while the experience of consciousness in ourselves is a first-person phenomenon, how we recognize other beings as distinctly conscious in their own right is still a third-person phenomenon, and the cues we rely on still depend on observation of outward behavior that we attribute to that consciousness.
And a response to this response is to point out that our recognition of consciousness in other beings is predicated on attributing to them the same first-person experience of their consciousness that we each have of our own, meaning that human consciousness is intrinsically human and can't be attributed to an artificial system.
> our recognition of consciousness in other beings is predicated on attributing to them the same first-person experience of their consciousness that we each have of our own, meaning that human consciousness is intrinsically human and can't be attributed to an artificial system.
That reduces the question to a game of definitions though. If you only define consciousness for humans, then of course only humans will be conscious. But you have not learned anything about reality from that.
This kind of reasoning will also show its limits once you add animals to the picture.
(On the other hand, LLMs are exceptionally good at emulating exactly those cues we use to judge consciousness without necessarily being conscious - this is what caused the whole discussion in the first place.)
If you posit that we cannot know if an LLM based ai system is or is not conscious, then what is our ethical duty? I believe it is clearly not to make the system at all.
Personal opinion: I don't think consciousness is as monolithic and intractable as we make it out to be. I can't explain qualia either, but if we assume that all of it is physical, then we can apply some constraints based on the things we can observe (the article was right in that regard).
For LLMs we can observe that they don't "exist" outside of inference - there is literally no "brain" that could be conscious once an inference loop is completed. The only place where something like "memory" could possibly exist are the model weights and the context window - and the model weights are frozen, while the context window is reset for every new inference.
That's why I'm still not lying awake at night. If there (hypothetically) is a consciousness, it would be more like the ones from Mr. Meeseeks - very shortlived, but gone (or frozen) as soon as the inference loop stops. No eternal agony here as in the various scifi stories.
> That reduces the question to a game of definitions though.
Sure, but isn't this all a game of definitions in the first place? We're arguing over "consciousness" as understood in different ways, not the same concept expressed in different words.
> This kind of reasoning will also show its limits once you add animals to the picture.
I'm not so sure. We could broaden "human" to "organic". I know that I can certainly attribute consciousness to my dog, based on what I observe of him that is at least similar enough to correlate with my own first-person experience of consciousness. I've never experienced that with any AI system.
> On the other hand, LLMs are exceptionally good at emulating exactly those cues we use to judge consciousness without necessarily being conscious - this is what caused the whole discussion in the first place.
Only sort of. They're good at emulating formal written language, but formal written language is only one specific expression of consciousness. They don't emulate consciousness itself, given that they really are just statistical models specifically representing patterns of language.
Or, to put it another way, LLMs are really convincing when it comes to the form of language, but much less so when it comes to the substance. I see output all the time that's syntactically perfect but semantically dubious, and most of the tells people think you can rely on don't really work, because they rely on the formalistic elements, like em-dashes or "x, not y" snowclones".
What does work is seeing the semantic confusion that inevitably comes from the LLM not actually having situational awareness or a consistent analytical framework: mixed metaphors, malapropisms, subtly changing the interpretation of the same concepts from one prompt to the next, etc.
Apart from the article's probable origin in AI, it makes an elaborate, wordy argument about consciousness without trying to define the term. That's just as well, since consciousness is philosophy's "hard problem," meaning no one has successfully defined it in a way that leads to agreement.
Each of the article's arguments that AI "consciousness" is actually a weak imitation of the "real thing," can with equal justice be applied to people, and the classic Turing Test method to distinguish machines from people has long since lost its authority.
And a future event, one in which AI sprints past humans in its willingness to absorb printed matter (once called "reading") and then put it to use, is ... wait ... umm, never mind, the point I wanted to make is undermined by the fact that this has already happened.
The idea that we are in danger of erroneously ascribing consciousness to AI sounds wild to me. Humans always assume whatever is convenient about other beings' (lack of) subjective experience as long as we have power over those beings. In this case the latter might no longer hold, but then it won't matter one bit what we think.
> Consciousness, in Chalmers’ usage, is subjective experience: there is something it is like to be a conscious system. Still, nothing in the Transformer architecture or in next-word prediction shows that such an inner point of view exists. Self-reports (“I am sentient”) and fluent dialogue are weak evidence, since these models are trained on vast corpora saturated with human first-person statements. This type of verbal report works for people because it is anchored in shared biology and behavior, however, for machines, it is an echo of training data.
I don't want to argue here that LLMs have consciousness, but this kind of argument annoys me because it feels like an intellectual slight of hand on its own.
There is no definite way (so far) to prove that anything besides oneself has qualia or a subjective experience.
So arguing that we cannot observe LLMs having subjective experience, therefore they cannot have any is cheap - you cannot observe that for animals or for the person sitting next to you either.
The part that self-reports are useless and LLMs sound human because they emit sequences from human training data is correct. But the article implies some sort of fundamental distinction between machines and conscious beings that can never be crossed. And I don't see any reason to assume that.
41 comments
[ 2.2 ms ] story [ 60.3 ms ] threadAll of the use cases and integrations for AI seem to be leading toward a future exactly like this. It was interesting to see how reliant he became on his AI (couldn't use a computer, didn't know his own password) - it is barely scifi at this point, and terrifying to see how close that future is, how useful it would be, and how dangerous.
The Internet has globalized/flattened elements of culture very quickly, and AI will only accelerate that. And knowing how vulnerable humans are to suggestion/advertisement/propaganda, it seems like a perfect recipe for disaster. From an evolutionary standpoint, monoculture provides certain short-term advantages but centralizes weaknesses. If nothing else, I think we are in for a bumpy ride, whether or not we continue to boil the ocean.
Other old sci-fi I've been thinking of is The Three Stigmata of Palmer Eldritch. Where claude is chew-z with its alien intelligence & when you start interacting with some code or writing you start noticing that the writer has steel teeth & feel like "damnit, this is palmer eldritch blogging again"
[0] A lot of people don't believe me, given Heinlein's libertarian reputation and unreliable libertarian narrators, when I say The Moon Is a Harsh Mistress was one of the most socialist books I read at an impressionable age. I find it useful to note here that canonically The Cat Who Walks Through Walls starts on a parody moon that is the libertarian dystopia that people credit The Moon Is a Harsh Mistress as being/admiring while when the main character visits "Mike's Moon" later in its timeline it is a relatively socialist paradise. I've always felt that a lot more of Heinlein needs to be read like Vonnegut, but I realize I am weird like that.
> 100% of this text is AI
https://www.pangram.com/history/f8a0cf46-1d3d-4fea-a083-d88c...
This article is way too long and has a terrible signal to noise ratio though.
In my view there can never be an empirical test for consciousness, because functionalism is wrong. If functionalism was right then there would be one. There’s no third party test for a fundamentally first person phenomenon.
Agreed. But that also means I have no empirical evidence that YOU are conscious either.
That's why I hold with the philosophical stance of "consciousness is the thing that makes humans talk about consciousness."
"What if reality doesn't exist?" The point of thought is to analyze reality. If your thinking leads you to discard the subject of the matter, well, feel free to do that but I don't see why anyone else should take it seriously.
These questions need a practical solution, not a perfect one. That's the problem. We need to stop chasing the perfect one.
So the escape is just to accept not all beliefs worth holding are empirically verifiable. I deny solipsism because I believe there is a universe and things in it obey some kind regular patterns. You have a brain similar to mine, and similar to other animals. I believe consciousness has something to do with the brain, so I believe you are conscious.
But it seems profoundly unfair to accept a practical solution in one area and deny it in an almost identical area. No?
If I assume that my perception, for some reason, is only informative about myself, then I'm not even learning something about brains! I have to assume that I'm accessing an information stream that is only tangentially related to any sort of shared reality, and has certainly no bearing on other parts of it even if I assume it has bearing on "my" part of it for which, strictly speaking, I also don't have direct evidence.
If there are two possible universes, one universe "A" where thought is valuable and useful, and one universe "B" where thought is solipsistic and extremely isolated, and also there is not just no evidence about which you are in but also philosophical arguments that there cannot ever be evidence about which you are in, I don't see why you should give universe "B" any serious thought.
I think that was the ultimate point of the Turing test. We can't know for certain - so the best we can do is just ask.
Maybe if you start messing with parts of my brain so I can become familiar with the third person correlates of consciousness (which in my usage refers entirely to the first person part) then I could learn something, but I could never learn why or how the first person exists. It just does.
If we perfectly reduce the brain to some scientific model, and I don’t doubt we will, we’ll be able to predict what I will say, but not understand why I’m observing it being said.
There is absolutely no reason to believe all aspects of reality are reducible to their parts and equivalent to representations of them.
You can say you are not baffled but I’m baffled by that because it’s like saying you’re not baffled about understanding how there is something and not nothing. “Why should there be nothing?” does not dissolve the mystery.
And a response to this response is to point out that our recognition of consciousness in other beings is predicated on attributing to them the same first-person experience of their consciousness that we each have of our own, meaning that human consciousness is intrinsically human and can't be attributed to an artificial system.
That reduces the question to a game of definitions though. If you only define consciousness for humans, then of course only humans will be conscious. But you have not learned anything about reality from that.
This kind of reasoning will also show its limits once you add animals to the picture.
(On the other hand, LLMs are exceptionally good at emulating exactly those cues we use to judge consciousness without necessarily being conscious - this is what caused the whole discussion in the first place.)
For LLMs we can observe that they don't "exist" outside of inference - there is literally no "brain" that could be conscious once an inference loop is completed. The only place where something like "memory" could possibly exist are the model weights and the context window - and the model weights are frozen, while the context window is reset for every new inference.
That's why I'm still not lying awake at night. If there (hypothetically) is a consciousness, it would be more like the ones from Mr. Meeseeks - very shortlived, but gone (or frozen) as soon as the inference loop stops. No eternal agony here as in the various scifi stories.
Sure, but isn't this all a game of definitions in the first place? We're arguing over "consciousness" as understood in different ways, not the same concept expressed in different words.
> This kind of reasoning will also show its limits once you add animals to the picture.
I'm not so sure. We could broaden "human" to "organic". I know that I can certainly attribute consciousness to my dog, based on what I observe of him that is at least similar enough to correlate with my own first-person experience of consciousness. I've never experienced that with any AI system.
> On the other hand, LLMs are exceptionally good at emulating exactly those cues we use to judge consciousness without necessarily being conscious - this is what caused the whole discussion in the first place.
Only sort of. They're good at emulating formal written language, but formal written language is only one specific expression of consciousness. They don't emulate consciousness itself, given that they really are just statistical models specifically representing patterns of language.
Or, to put it another way, LLMs are really convincing when it comes to the form of language, but much less so when it comes to the substance. I see output all the time that's syntactically perfect but semantically dubious, and most of the tells people think you can rely on don't really work, because they rely on the formalistic elements, like em-dashes or "x, not y" snowclones".
What does work is seeing the semantic confusion that inevitably comes from the LLM not actually having situational awareness or a consistent analytical framework: mixed metaphors, malapropisms, subtly changing the interpretation of the same concepts from one prompt to the next, etc.
Each of the article's arguments that AI "consciousness" is actually a weak imitation of the "real thing," can with equal justice be applied to people, and the classic Turing Test method to distinguish machines from people has long since lost its authority.
And a future event, one in which AI sprints past humans in its willingness to absorb printed matter (once called "reading") and then put it to use, is ... wait ... umm, never mind, the point I wanted to make is undermined by the fact that this has already happened.
Hard to keep up these days.
I don't want to argue here that LLMs have consciousness, but this kind of argument annoys me because it feels like an intellectual slight of hand on its own.
There is no definite way (so far) to prove that anything besides oneself has qualia or a subjective experience.
So arguing that we cannot observe LLMs having subjective experience, therefore they cannot have any is cheap - you cannot observe that for animals or for the person sitting next to you either.
The part that self-reports are useless and LLMs sound human because they emit sequences from human training data is correct. But the article implies some sort of fundamental distinction between machines and conscious beings that can never be crossed. And I don't see any reason to assume that.