Doesn't this still presume that we understand our own consciousness, in order to make the comparison.
Where does our survival instinct come from? And why couldn't AI have one?
>>>Additional
Also, reproduction.
Humans are basically just Food, Sex, Survival. And consciousness is just a rule set for fulfilling those goals. So if a NN, modeled on US, does develop the same rules, why can't it have the same degree of consciousness. Who says we are consciousness?
This is one of those papers that uses a lot of big words to paper over the fact that it's really a philosophical opinion rather than a logical argument.
I've attempted desperately to understand this paper after thoroughly reading it and have made 0 progress. Can anyone who does understand it attempt to explain?
Currently my understanding is that this paper is claiming that "concepts" are a fundamental building block of experience (which relates to consciousness), and can only be built by a mapmaker which is something that directly converts continuous physical phenomena into discrete tokens. But I couldn't get further into how that related to consciousness.
EDIT: the paper seems to be assuming that something simulating a mapmaker, or the process of doing it, can by nature not be a mapmaker since performing alphabetization is inherently something that must be "instantiated". How do they confirm if something is doing simulation vs if it's actually instantiating it? How can you tell the difference? They say how, much like simulating photosynthesis will not produce glucose, simulating mapmaking won't produce concepts. But you can't measure concepts, they're intangible, so you can't differentiate simulating mapmaking vs a real mapmaker.
the abstract very directly and literally denies the titular claim. It states:
> [consciousness] requires active, experiencing cognitive agent to alphabetize continuous physics into a finite set of meaningful states.
This may well be true—I think it is.
I also think that it is both widely understood and self-evident that the most promising path to machine consciousness, is via AI with continuous sensory input and agency, of which "world models" are getting a lot of attention.
When an AI system has phenomenology, the goal posts are going to start to resemble the God of the Gaps; at some point, critics will be arguing with systems which have a world model, a self model, agency, and literally and intrinsically understand the world not simply as symbolic tokens, but as symbolic tokens which are innately coupled to multi-modal representations of the things represented.
In other words, they will look—and increasingly, sound—a lot like us.
It's not that any of this is easy, nor that there is some paricular timeline, but it increasingly looks like "a mere question of engineering," and not blocked by fundamentals. It's blocked by the cost of computation and the limitations of our current model topologies.
But HN readers well know that the research frontier is far ahead of commercialized LLM, and moving fast.
An interesting time to be an agent with a phenomenology, is it not?
Yawn. We have no understanding of what consciousness actually is. Therefore attempting to prove whether a system can or cannot be conscious is something we can't prove or disprove at this point.
I think this is a circular argument. It defines a separation between computation and experience (between the abstraction and the "mapmaker") and then concludes that computation cannot be experience because they are in separate categories.
There are really only two solutions to the Hard Problem of Consciousness:
1. Consciousness is an unknown physical something (force/particle/quantum whatever).
2. Consciousness is an illusion. It is the software telling itself something.
[Some people would add "3. Consciousness is an emergent property of certain systems." But that just raises the question of what emerged? Is it a physical structure, like a tornado (also an emergent property) or an internal feedback loop (i.e., an illusion).]
The problem with #1 is that it's hard to cross the chasm from non-conscious to conscious with a bucket of parts. How is it that atoms/electrons/photons suddenly start experiencing pain? What is it, in terms of atoms/forces, that's experiencing the pain?
#2 makes more sense. Pain isn't a real thing any more than an IEEE float is a real thing. A circuit flips bits and an LED shows a number. A set of neurons fire in a pattern and the word "Ow!" comes out of someone's mouth.
I've never understood why certain philosophers view computation as some kind of abstract symbolic manipulation, while they easily accept that consciousness is a physical process.
Computation is something that a computer provably does. We build physical hardware, at great effort, to do computation. The hardware works and does the computation regardless of whether there is anyone to understand or interpret it. If it didn't, we couldn't have built anything like, say, an automatic door: that is a form of computation that provably happens as a physical process that is completely observer-independent.
Sure, a different entity than a human might view it completely differently than a door opening when someone is near - but the measurable physical effect would be the exact same, with the exact same change in momentum and position of the atoms in what we call the door based on the relative position of some other atoms and the sensor.
>Computation is something that a computer provably does.
This is a circular definition. In order to properly define the concept, we must be able to word it without using "computing devices" in the definition.
Finding a satisfactory definition for what constitutes a "Computation" is actually an interesting debate goes back to the 1600s. Currently, the mainstream definition (from wikipedia) gives that: "A computation is any type of arithmetic or non-arithmetic calculation that is well-defined".
One way to understand the author is to learn more about the "The mapping account" theory behind computation: "a physical system can be said to perform a specific computation when there is a mapping between the state of that system and the computation such that the 'microphysical states [of the system] mirror the state transitions between the computational states.'"
I wasn't trying to use that as a definition of what computation means. My point is that regardless of how we define it, the fact is that the devices we build to do some form of computation are objectively successful at doing what they were built to do.
For example, you can say that Firefox is a human-centric abstraction, and that my computer isn't running Firefox right now in an objective sense, that this is is just a human-centric interpretation of what the physical device is doing, and that there exist other computations that we could assign to it.
But what you can't say is that the device is not affecting the physical world in ways that are consistent with performing the Firefox computation, such as causing certain specific wavelengths of light to be emitted by the screen based on state that is stored in a server in the YCombinator data center. This is a measurable fact of the physical world that is independent of the model of computation you chose to ascribe to the physical device - any consistent mapping will have to preserve this same physical property.
The paper addresses this point in section 3.2. They aren't debating the fact that a physical process is taking place in a computer running a program. They are arguing that the semantic interpretation of the output of that program is indeterminate and dependent on the mapping function:
> A single physical vehicle (bottom) possesses a fixed causal trajectory. However, it does not instantiate a unique computation. Depending on the alphabetization key applied (fA or fB ), the same physical states can be mapped to entirely different abstract computations (Top Left vs. Top Right). Therefore, computation cannot be intrinsic to the physics (p).
So yes there is a physical process generating your Firefox browser, but there is also a mapping function taking that program and interpreting that it should display your Firefox browser. There are any number of mapping functions that could be applied to the physical state in order to display other things on your screen besides the browser. Therefore, the Firefox browser being displayed is not inherent or intrinsic to the physical state of your computer. If we did not have the right mapping function, we would have no way of knowing or inferring or discovering which mapping function is correct.
But a robot doing closed loop RL in the world is its own mapmaker, no? I feel like you'd need to answer: At what point does a system whose representations are shaped by its own causal history with the world, stop counting as a mere simulation..?
Consciousness is an engineering problem not a philosophical one. How do you get a tiny fraction of the many billion experiences that cohere to create your self to listen to, and decide what sensory data to turn into your next experience?
The engineering problem is that this decentralised moment to moment consensus has to span the galactic distance of your mind (from the perspective of a neuron) and do it fast and cheap (on a tiny metabolic budget)
You might like our book Journey of the Mind if you'd rather skip the onerous philosophical jargon and get a systems neuroscience perspective
If I understand the paper correctly, it does not really argue against highly capable general AI. It argues against conflating capability with phenomenology.
That makes me wonder whether “AGI” is doing too much work as a term. In common usage it often evokes something like HAL 9000: a capable system that is also a subject. But the paper seems compatible with a future of very general, very useful AI systems that are not conscious subjects at all.
I would argue that, before we can begin to address whether or not AI can instantiate consciousness, we should agree on a practical, unequivocal definition of what consciousness is... and I think we're still pretty far from that milestone... Until then, this kind of argument are nothing more than pipe dreams, solipsism, and idle philosophising
I had the same initial thought but to be fair the paper addresses this explicitly and the author believes their argument can hold without fully understanding consciousness
If we agree that consciousness is a physical process part of our universe, I think the better and simpler question is whether or not computers can simulate any physical process. Currently quantum processes might still be a frontier but quantum computers and their hardware should allow us to simulate them.
If we can simulate any physical process, it then becomes more philosophical in my opinion. Whether the simulation is the same as the real thing even though it is exactly the same. It becomes the same kind of question then for example whether or not your teleported self is still you after having been dematerialized and rematerialized from different atoms. The answer might be no, but you rematerialized self still definitely thinks it is yourself.
Pretty crazy how the author's 10+ years of academic research on computational neuroscience + 14 years with DeepMind is not enough to make claims in this topic, but hacker news commentators know better after quickly skimming the abstract. This was barely posted ~30 minutes ago and yet commentators are just outright dismissing it based on their own (probably) incorrect interpretation of the paper based on the title and abstract.
If I understand this correctly based on a quick read, it argues that subjective experience arises at the (or in the) "alphabetization" process where continuous physical states (e.g. voltage) are mapped to discrete logical states (roughly like e.g. a bit) or "concepts" (figure 2).
Per this reading, implementing something in ASIC would make it have (a different) experience, as opposed to CPU/GPU. Not sure what would be the case for FPGAs.
It also seems to rely on the classical "GOFAI" idea of symbol manipulation, and e.g. denies experience that isn't discretizable into concepts. Or at least the system producing such concepts seems to be necessary, not sure if some "non-conceptual experiences" could form in the alphabetization process.
It reads a bit like a more rigorous formulation of the Searle's "biological naturalism" thesis, the central idea being that experience can not be explained at the logical level (e.g. porting an exact same algorithm to a different substrate wouldn't bring the experience along in the process).
Bold title for something from DeepMind. I thought a crank submission slipped onto the front page somehow. I guess the next paper will be “Why AI cannot instantiate God”?
One of her points is that there are various pesky consequences for AI companies if AI becomes to be seen as conscious, such as what the paper calls the "welfare trap": if AI systems are widely regarded as being conscious or sentient, they will be seen as "moral patients", reinforcing existing concerns over whether they are being treated appropriately. This paper explicitly says that its conclusion "pulls the field of AI safety out of the welfare trap, [allowing] us to focus entirely on the concrete risks of anthropomorphism [by] treating AGI as a powerful but inherently non-sentient tool."
47 comments
[ 4.6 ms ] story [ 65.4 ms ] threadWhere does our survival instinct come from? And why couldn't AI have one?
>>>Additional
Also, reproduction. Humans are basically just Food, Sex, Survival. And consciousness is just a rule set for fulfilling those goals. So if a NN, modeled on US, does develop the same rules, why can't it have the same degree of consciousness. Who says we are consciousness?
(That one didn't make the frontpage, so we won't treat it as a dupe. - https://news.ycombinator.com/newsfaq.html)
Currently my understanding is that this paper is claiming that "concepts" are a fundamental building block of experience (which relates to consciousness), and can only be built by a mapmaker which is something that directly converts continuous physical phenomena into discrete tokens. But I couldn't get further into how that related to consciousness.
EDIT: the paper seems to be assuming that something simulating a mapmaker, or the process of doing it, can by nature not be a mapmaker since performing alphabetization is inherently something that must be "instantiated". How do they confirm if something is doing simulation vs if it's actually instantiating it? How can you tell the difference? They say how, much like simulating photosynthesis will not produce glucose, simulating mapmaking won't produce concepts. But you can't measure concepts, they're intangible, so you can't differentiate simulating mapmaking vs a real mapmaker.
the abstract very directly and literally denies the titular claim. It states:
> [consciousness] requires active, experiencing cognitive agent to alphabetize continuous physics into a finite set of meaningful states.
This may well be true—I think it is.
I also think that it is both widely understood and self-evident that the most promising path to machine consciousness, is via AI with continuous sensory input and agency, of which "world models" are getting a lot of attention.
When an AI system has phenomenology, the goal posts are going to start to resemble the God of the Gaps; at some point, critics will be arguing with systems which have a world model, a self model, agency, and literally and intrinsically understand the world not simply as symbolic tokens, but as symbolic tokens which are innately coupled to multi-modal representations of the things represented.
In other words, they will look—and increasingly, sound—a lot like us.
It's not that any of this is easy, nor that there is some paricular timeline, but it increasingly looks like "a mere question of engineering," and not blocked by fundamentals. It's blocked by the cost of computation and the limitations of our current model topologies.
But HN readers well know that the research frontier is far ahead of commercialized LLM, and moving fast.
An interesting time to be an agent with a phenomenology, is it not?
I've found this one (which makes no falsification claims about computers re consciousness) to be an interesting read: https://arxiv.org/pdf/2409.14545
There are really only two solutions to the Hard Problem of Consciousness:
1. Consciousness is an unknown physical something (force/particle/quantum whatever). 2. Consciousness is an illusion. It is the software telling itself something.
[Some people would add "3. Consciousness is an emergent property of certain systems." But that just raises the question of what emerged? Is it a physical structure, like a tornado (also an emergent property) or an internal feedback loop (i.e., an illusion).]
The problem with #1 is that it's hard to cross the chasm from non-conscious to conscious with a bucket of parts. How is it that atoms/electrons/photons suddenly start experiencing pain? What is it, in terms of atoms/forces, that's experiencing the pain?
#2 makes more sense. Pain isn't a real thing any more than an IEEE float is a real thing. A circuit flips bits and an LED shows a number. A set of neurons fire in a pattern and the word "Ow!" comes out of someone's mouth.
Computation is something that a computer provably does. We build physical hardware, at great effort, to do computation. The hardware works and does the computation regardless of whether there is anyone to understand or interpret it. If it didn't, we couldn't have built anything like, say, an automatic door: that is a form of computation that provably happens as a physical process that is completely observer-independent.
Sure, a different entity than a human might view it completely differently than a door opening when someone is near - but the measurable physical effect would be the exact same, with the exact same change in momentum and position of the atoms in what we call the door based on the relative position of some other atoms and the sensor.
This is a circular definition. In order to properly define the concept, we must be able to word it without using "computing devices" in the definition.
Finding a satisfactory definition for what constitutes a "Computation" is actually an interesting debate goes back to the 1600s. Currently, the mainstream definition (from wikipedia) gives that: "A computation is any type of arithmetic or non-arithmetic calculation that is well-defined".
One way to understand the author is to learn more about the "The mapping account" theory behind computation: "a physical system can be said to perform a specific computation when there is a mapping between the state of that system and the computation such that the 'microphysical states [of the system] mirror the state transitions between the computational states.'"
For example, you can say that Firefox is a human-centric abstraction, and that my computer isn't running Firefox right now in an objective sense, that this is is just a human-centric interpretation of what the physical device is doing, and that there exist other computations that we could assign to it.
But what you can't say is that the device is not affecting the physical world in ways that are consistent with performing the Firefox computation, such as causing certain specific wavelengths of light to be emitted by the screen based on state that is stored in a server in the YCombinator data center. This is a measurable fact of the physical world that is independent of the model of computation you chose to ascribe to the physical device - any consistent mapping will have to preserve this same physical property.
> A single physical vehicle (bottom) possesses a fixed causal trajectory. However, it does not instantiate a unique computation. Depending on the alphabetization key applied (fA or fB ), the same physical states can be mapped to entirely different abstract computations (Top Left vs. Top Right). Therefore, computation cannot be intrinsic to the physics (p).
So yes there is a physical process generating your Firefox browser, but there is also a mapping function taking that program and interpreting that it should display your Firefox browser. There are any number of mapping functions that could be applied to the physical state in order to display other things on your screen besides the browser. Therefore, the Firefox browser being displayed is not inherent or intrinsic to the physical state of your computer. If we did not have the right mapping function, we would have no way of knowing or inferring or discovering which mapping function is correct.
The engineering problem is that this decentralised moment to moment consensus has to span the galactic distance of your mind (from the perspective of a neuron) and do it fast and cheap (on a tiny metabolic budget)
You might like our book Journey of the Mind if you'd rather skip the onerous philosophical jargon and get a systems neuroscience perspective
https://saigaddam.medium.com/consciousness-is-a-consensus-me...
That makes me wonder whether “AGI” is doing too much work as a term. In common usage it often evokes something like HAL 9000: a capable system that is also a subject. But the paper seems compatible with a future of very general, very useful AI systems that are not conscious subjects at all.
If we can simulate any physical process, it then becomes more philosophical in my opinion. Whether the simulation is the same as the real thing even though it is exactly the same. It becomes the same kind of question then for example whether or not your teleported self is still you after having been dematerialized and rematerialized from different atoms. The answer might be no, but you rematerialized self still definitely thinks it is yourself.
Per this reading, implementing something in ASIC would make it have (a different) experience, as opposed to CPU/GPU. Not sure what would be the case for FPGAs.
It also seems to rely on the classical "GOFAI" idea of symbol manipulation, and e.g. denies experience that isn't discretizable into concepts. Or at least the system producing such concepts seems to be necessary, not sure if some "non-conceptual experiences" could form in the alphabetization process.
It reads a bit like a more rigorous formulation of the Searle's "biological naturalism" thesis, the central idea being that experience can not be explained at the logical level (e.g. porting an exact same algorithm to a different substrate wouldn't bring the experience along in the process).
One of her points is that there are various pesky consequences for AI companies if AI becomes to be seen as conscious, such as what the paper calls the "welfare trap": if AI systems are widely regarded as being conscious or sentient, they will be seen as "moral patients", reinforcing existing concerns over whether they are being treated appropriately. This paper explicitly says that its conclusion "pulls the field of AI safety out of the welfare trap, [allowing] us to focus entirely on the concrete risks of anthropomorphism [by] treating AGI as a powerful but inherently non-sentient tool."