Seriously, why won't this 2+ decade old argument die? It's sophomoric and picked apart in second year cognitive systems classes by 19 year old children. It glaringly misunderstands and misrepresents. If people are genuinely interested in AI they should take a basic course in it at university or pick up a real book on the subject.
In my opinion, this argument have existential consequences: either it's not possible to emulate mind on turing machine or we don't have qualia and consciousness (it's just the illusion, that we have). And i don't like the latter.
"What's the difference between having an illusion of consciousness and having the real thing?"
Primarily, the word "illusion".
At this point, having read up on these issues for a while, anytime I see the word "illusion" now I tend to just shut the book/leave the webpage/whatever. Have you ever, once, seen someone sit down and say what they mean by "illusion", or give a way of telling you how to distinguish between "illusion" or "reality"? Maybe you have (no sarcasm intended), but I've never seen it, and without that the word basically just marks someone who is trying to sound insightful without doing the hard work of actually being insightful and saying something without leaving an enormous linguistic void right in the center of the argument.
Whether consciousness is only illusion is of no consequence. This path of thoughts (just like Berkeley's subjective idealism or its modern form, the Matrix movie) ain't leading anywhere (besides madness) hence it can be classified as solipsism.
In my opinion, what matters that you experience consciousness, life, happiness or sadness. Have great meals, love the people who love you, do cool stuff as a 'hacker'. Your life will be as 'real' as it could get, whether it's an illusion or not.
It's not just your opinion ;) This argument is about the nature of consciousness more than anything. It fights for a "subjective ontology" of the mental and is not easily dismissed as the earlier commentator would like to believe. The rapidity of upvotes and apparent sympathy to that claim by members of this community is certainly understandable given that the stance is anti-verificationist and presents problems for a purely scientific explanation of consciousness, but imho at our current level of intelligence/intellectual evolution a purely scientific explanation is horribly dehumanizing and insufficient.
Take a look at "Society of mind" by Marvin Minsky. We already understand pretty well that the mind is a colony of mostly independent system, working in various configurations. Here's the first chapter: http://www.edge.org/3rd_culture/minsky07/minsky07_index.html
As far as I'm concerned, qualia is just a fancy synonym for "soul". I don't miss it one bit. Consciousness on the other hand, that we have. Only it turns out to be a more elusive concept then we once thought.
According to the "Chinese room" argument, a machine is not capable of genuinely understanding a language (since it is only looking up and applying transformation rules). It is generally believed that a human can, so that would mean a human mind cannot be fully emulated by a Turing machine. The only alternative seems to be that a human actually has the same limitation as a Turing machine (non-consciousness) and does not really understand language either.
> The purpose of philosophy is to protect us from bad philosophy.
Nicely put. I like Wittgenstein's way of putting it as well, "What is your aim in philosophy?--To shew the fly the way out of the fly-bottle." (Philosophical Investigations 309)
it won't "die" for the same reason that it's still used at university - both because it raise some interesting issues and because it played an important part in the historical debate over ai.
also, if you actually read the link you'll see that it's placed in context, with the "systems reply" explained in detail.
To a software person who's given AI some thought, the holes in Searle's argument are glaring. I had a very awkward conversation in college with a philosophy student who had just been taught it and was taking it very seriously; I guess they have a lot of reverence for Great Thinkers who hand down Deep Insights, whereas to me this guy Searle sounded like an imbecile.
Despite my initial horror at someone's taking the Chinese Room argument at face value, I realize it raises interesting questions about the chasm between AI's starting point (e.g. playing chess with S-expressions) and real human-like intelligence. I think we have more perspective on the problem these days for countless reasons -- work on understanding how subsystems in the brain work, being able to build or simulate neuron-like systems, applying powerful computers to a wide variety of problems.
The big questions that remain are ones like whether a human brain can be simulated; how brain-like a machine would have to be to "understand" things -- self-aware? neural-network-like? embodied? -- and to what extent constructs like language reflect deep structure of the mind (i.e. human firmware we have to reverse engineer and duplicate) rather than universal logical constructs.
>I guess they have a lot of reverence for Great Thinkers who hand down Deep Insights, whereas to me this guy Searle sounded like an imbecile.
Have you ever read the Chinese room article? It's pretty clear if you read it that Searle isn't an imbecile. The vast majority of "refutations" of his argument simply miss his point entirely. (Which is basically, "Why on Earth would you think that merely performing the same computations as a human brain could cause conscious experiences?")
> I guess they have a lot of reverence for Great Thinkers who hand down Deep Insights
Really? I've never met any philosophy students who fit that description.
I've never been able to figure out how this (and most other philosophical arguments against AI) don't apply just as well to individual neurons. Though I suppose if one wants to frame it as an argument that neither computers nor human brains are capable of intelligence, I might be persuaded.
In fact, for sake of argument, I claim that I am, in fact, just an elaborate system of symbolic manipulation with no actual comprehension or conscious experience, a bunch of meaningless neural impulses with no greater understanding of English than the "Chinese Room" understands Chinese; and I invite anyone to attempt to persuade me otherwise.
I think the fact that the argument is equally applicable to the human brain gets at the heart of the general problem of consciousness, of which AI is just a special case. Consciousness is not an observable phenomenon (excepting one's experience of one's own consciousness), so it seems unlikely to me that we can ever understand it in terms of observable phenomena-- i.e. as a function/result of things like neurons or transistors.
I think you could make the same argument for lots of things computers do. Could Grand Theft Auto 4, including the character AI and per-pixel 3D rendering, be implemented by people following instructions on cards? Yes, because Turing machines yadda yadda. But it's inconceivable to non-programmers. The Chinese room argument is convincing for the same reason: doing something AI-like requires billions of steps and non-programmers can't imagine building up something that complex from primitive operations.
Searle's problem is that, much as we can't imagine how silicon can be conscious, we can't imagine how squishy physical brains are capable of conscious experiences either, even though we know they are, because we are. His solution is that biological stuff just has these special properties, that's a "brute fact" about biological stuff and it's not true of silicon stuff. And I think most philosophers of mind (who are, by and large, nonprogrammers) think that's too 'magical' to be true.
In fact a very popular idea now in philosophy is that consciousness is (in some sense) a fundamental physical property, and when physical things are arranged in the right way, they become conscious like us or any other animal. And the proper arrangement of particles to invoke these high-level conscious properties must help organisms survive, or else it'd be unlikely that we have them. And so machines structured the right way can probably be conscious too.
I don't understand why people are willing to even accept the premise. The system as described (a book of instructions for manipulating Chinese symbols) can't usefully answer the question "what time is it?"; why should I believe it could carry on a lucid-but-very-slow conversation in Chinese?
Imagine that also in the room is a triangle with four sides. Now the Chinese Room Argument disproves AI and geometry! What subject do you want to demolish next?
I think this is the only real objection to Searle.
People get handwavey in these thought experiments, that we can just assume that such a system can be built. Well, no, we can't, not even in theory. It's trivial to show that a Chinese Room that could converse for more than 15 minutes would have to contain more elementary particles than the entire planet.
There is an analogy here with Chomsky's linguistics. You can disagree with the specifics of his theory, but he did show that there was no way that people were learning sentences like parrots and repeating them. There had to be some computational / grammatical process.
Anybody can imagine mechanical alternatives for doing any information processing. To an extent, that drives the whole of the programming industry. The whole question of intelligence is that we can do it with a few pounds of biological material. Anything that doesn't address this is missing the point.
" It's trivial to show that a Chinese Room that could converse for more than 15 minutes would have to contain more elementary particles than the entire planet."
I still don't believe you. So the numbers are made up? "15 minutes" -> "entire planet"? How much for 10 minutes? And for 1 minute? Is that obvious too?
So you think it's very obvious that it's impossible to build a chatbot that speaks chinese for 15 minutes in a computer like the one I'm using now (even one that claims to be a child for example). And why couldn't we just clone a Chinese person brain and say it's a computer? Why do you think a brain snapshot would require the size of a planet?
Ah, sorry, the "more elementary particles than the entire planet" claim wasn't made by me. My first comment was too vague, here is another try:
Yes, it sounds like a very strange claim, since isn't biological material made from elementary particles, and clearly you could fit someone who speaks Chinese into the room. Obviously that assumes that people are made of elementary particles, but if someone is going to argue that, I'm going to need a better definition of elementary particles to make any progress.
I've read up a little more on the Chinese Room and I retract my argument. I misunderstood the idea.
I thought Searle was asserting that a human mind can be simulated by a dumb program that only provided cached responses. It's easy to show there are more than 10e50 possible conversational paths in a 15 minute conversation (even a chess game has that many, and conversation is more complex than chess), and there are only 10e50 atoms on earth.
But it seems Searle's real argument was that the man inside the box is performing all the actions of an artificially intelligent computer program, just with pencil and paper. Humans clearly perform such actions with just a few pounds of brain matter. Even if the Chinese Room operator requires continent-wide resources, such difficulties don't address the essence of Searle's argument.
He seems to be asserting there's something special about the brain called "intentionality" which is unlike a computer running an algorithm. He might be right, but there seems to be no evidence for such a thing, or any way to prove that an entity which claimed "intentionality" didn't really have it.
What's wrong with the objection that, if you accept that any two universal Turing machines are equivalent in power, Searle's argument amounts to(letting P be "there exists an algorithm A such that any computing system executing A understand Chinese" and Q be "a human simulating a computing system executing the algorithm A understands Chinese") "Suppose P. It's obvious that ¬Q. Since P is equivalent to Q, P leads to a contradiction. Therefore ¬P"? It begs the question.
Searle's thought experiment does inspire one of my own. Consider a computer that executes a sorting algorithm(mergesort, for example) on inputs consisting of sequences of number. Does it understand how to sort a sequence of numbers? What if a person uses mergesort manually to sort sequences of numbers? Does this person understand how to sort a sequence of numbers? Is there any fundamental difference between the understanding of a person who uses mergesort to sort sequences of numbers and that of a computer that uses mergesort to sort sequences of numbers?
Searle doesn't make an argument about formally equivalent Turing machines, he makes an argument about a person using an instruction book to manipulate strange characters.
It's a perfect trap for your intuition. You have an intuitive feeling that if you were to have a conversation in Chinese by following mechanical instructions from a book, you still wouldn't "really understand" Chinese. All well and good, so far.
But there are enormous leaps from the Searle-scenario that's so appealing to your intuition to a machine powerful enough to hold a conversation. As a trivial example, the Chinese room cannot even simulate a Turing machine unless you give the person inside rather a lot of scratch paper.
You can keep countering objections like this by adding more stuff to the Room. So to hold a conversation about the weather, we'll need to add some separate input channels to observe the outside world, and to talk about the Room's plans for today, we'll need to throw in something that can keep track of time, .... it just goes on. (And while you're adding all the extra machinery, don't forget that even the original book is incalculably large, not the merely hefty tome that you're picturing in your mind's eye.)
As you do this, you get farther and farther away from the original appeal to your intuition. So far that your only hope of making sense is to acknowledge that your intuitions about the simplified situation has absolutely nothing to do with a Chinese Room powerful enough to actually meet the original stipulation, that the room can "pass the Turing Test for understanding Chinese".
Searle doesn't make an argument about formally equivalent Turing machines, he makes an argument about a person using an instruction book to manipulate strange characters.
Right, this discussion is imcomplete without bringing in Church's thesis.
* So to hold a conversation about the weather, we'll need to add some separate input channels to observe the outside world, and to talk about the Room's plans for today, we'll need to throw in something that can keep track of time, .... it just goes on.*
Or these could be predefined and static.
(And while you're adding all the extra machinery, don't forget that even the original book is incalculably large, not the merely hefty tome that you're picturing in your mind's eye.)
This is like objecting that a Turing machine would take too many steps to solve a sufficiently large instance of some problem (e.g. more than there are Planck times in the age of the universe). The problem is still Turing-computable.
"Searle doesn't make an argument about formally equivalent Turing machines, he makes an argument about a person using an instruction book to manipulate strange characters."
In what way is a hypothetical person using "a book of instructions for manipulating the symbols (the program)" different from a universal Turing machine executing the same instructions? When Turing manually executed his computer chess program, how did this differ from having a computer execute it?
"It's a perfect trap for your intuition. You have an intuitive feeling that if you were to have a conversation in Chinese by following mechanical instructions from a book, you still wouldn't "really understand" Chinese."
I fail to understand how this portion of what you're saying disagrees at all with what I said. I don't really see the rest of what you say as disagreeing with what I said, but it is at least not identical to it.
Thinking out loud here I guess, but I'd like to hear Eliezer's opinion on the interpretations of QM that do reconcile conscious experience with these probabilities in a way that doesn't require splitting worlds, like these: http://arxiv.org/pdf/quant-ph/0603027
Richard Gabriel has a nice write-up on this, well worth reading especially if you're in the sputtering 'zomg this is SO dumb' camp. It's also a much quicker read than the entirety of the Stanford page.
> Most of the critics of Searle's argument fail to respond directly to his points. This is actually not all that surprising in that most of the critics are scientists and Searle's argument is philosophical, but it means that doing a point by point critique of the argument is unlikely to be very useful.
Well now, that says something about philosophy's relevance to science.
> Many people consider the concept of gender inseparable from its fleshly and biological origin and nature.
And many people don't. The argument is just philosophical, what's the big deal?
I have a grudge against written philosophy. The meaning of words is subjective. It is defined by what relations you draw in your brains when you read or hear that word. This not only makes words quite unimportant, they turn your mind into a philosophical mine field. It is constantly under the assumption that words have some universal meaning. People seeking for the "intelligence" or "truth" that they have in their head, but they're constantly redefining it based on new insights obtained in their search. As a result, all these concepts seem unattainable.
If you take the focus off the word, much of your prejudice disappears. You see the relationships, the structure, the observations, the logic, the nuance. You can see you used two distinct meanings of the word "intelligence". One is a set of expected reactions, the other is your consciousness. Now this gives rise to a logical question: Are these two things the same? The Chinese room shows that's not necessarily the case. However, might that feeling you call consciousness be a side-effect of a the particular type of Chinese room that's going on in your head? That's how you get interesting philosophy.
No. Or at least, not entirely, not in any significant way, and mostly just no.
If language were truly subjective, we should expect almost all attempts at communication to fail. If language were radically subjective, then two people speaking to each other in the same language would be for all intents and purposes like a speaker of (only) English trying to communicate to a speaker of (only) Chinese. It's manifestly the case that this isn't so. Beyond this, if language were truly subjective, we wouldn't be able to think (since the person-to-person communication problem would emerge even for one person).
45 comments
[ 4.0 ms ] story [ 97.9 ms ] threadIt's the room that is conscious not the person. Of course the person can't speak Chinese. Neither can the neuron in my brain.
An illusion experienced by whom?
What's the difference between having an illusion of consciousness and having the real thing?
Primarily, the word "illusion".
At this point, having read up on these issues for a while, anytime I see the word "illusion" now I tend to just shut the book/leave the webpage/whatever. Have you ever, once, seen someone sit down and say what they mean by "illusion", or give a way of telling you how to distinguish between "illusion" or "reality"? Maybe you have (no sarcasm intended), but I've never seen it, and without that the word basically just marks someone who is trying to sound insightful without doing the hard work of actually being insightful and saying something without leaving an enormous linguistic void right in the center of the argument.
In my opinion, what matters that you experience consciousness, life, happiness or sadness. Have great meals, love the people who love you, do cool stuff as a 'hacker'. Your life will be as 'real' as it could get, whether it's an illusion or not.
As far as I'm concerned, qualia is just a fancy synonym for "soul". I don't miss it one bit. Consciousness on the other hand, that we have. Only it turns out to be a more elusive concept then we once thought.
(1) It is possible to emulate the mind on a Turing Machine.
to:
(2) We don't have qualia or consciousness.
In my opinion, there's nothing about (1) that entails (2).
The purpose of philosophy is to protect us from bad philosophy.
Nicely put. I like Wittgenstein's way of putting it as well, "What is your aim in philosophy?--To shew the fly the way out of the fly-bottle." (Philosophical Investigations 309)
So you prefér circular argumentation?
also, if you actually read the link you'll see that it's placed in context, with the "systems reply" explained in detail.
To a software person who's given AI some thought, the holes in Searle's argument are glaring. I had a very awkward conversation in college with a philosophy student who had just been taught it and was taking it very seriously; I guess they have a lot of reverence for Great Thinkers who hand down Deep Insights, whereas to me this guy Searle sounded like an imbecile.
Despite my initial horror at someone's taking the Chinese Room argument at face value, I realize it raises interesting questions about the chasm between AI's starting point (e.g. playing chess with S-expressions) and real human-like intelligence. I think we have more perspective on the problem these days for countless reasons -- work on understanding how subsystems in the brain work, being able to build or simulate neuron-like systems, applying powerful computers to a wide variety of problems.
The big questions that remain are ones like whether a human brain can be simulated; how brain-like a machine would have to be to "understand" things -- self-aware? neural-network-like? embodied? -- and to what extent constructs like language reflect deep structure of the mind (i.e. human firmware we have to reverse engineer and duplicate) rather than universal logical constructs.
Have you ever read the Chinese room article? It's pretty clear if you read it that Searle isn't an imbecile. The vast majority of "refutations" of his argument simply miss his point entirely. (Which is basically, "Why on Earth would you think that merely performing the same computations as a human brain could cause conscious experiences?")
> I guess they have a lot of reverence for Great Thinkers who hand down Deep Insights
Really? I've never met any philosophy students who fit that description.
In fact, for sake of argument, I claim that I am, in fact, just an elaborate system of symbolic manipulation with no actual comprehension or conscious experience, a bunch of meaningless neural impulses with no greater understanding of English than the "Chinese Room" understands Chinese; and I invite anyone to attempt to persuade me otherwise.
we don't know
The man in the room is the neuron it's the house that is the brain.
Searle's problem is that, much as we can't imagine how silicon can be conscious, we can't imagine how squishy physical brains are capable of conscious experiences either, even though we know they are, because we are. His solution is that biological stuff just has these special properties, that's a "brute fact" about biological stuff and it's not true of silicon stuff. And I think most philosophers of mind (who are, by and large, nonprogrammers) think that's too 'magical' to be true.
In fact a very popular idea now in philosophy is that consciousness is (in some sense) a fundamental physical property, and when physical things are arranged in the right way, they become conscious like us or any other animal. And the proper arrangement of particles to invoke these high-level conscious properties must help organisms survive, or else it'd be unlikely that we have them. And so machines structured the right way can probably be conscious too.
Imagine that also in the room is a triangle with four sides. Now the Chinese Room Argument disproves AI and geometry! What subject do you want to demolish next?
People get handwavey in these thought experiments, that we can just assume that such a system can be built. Well, no, we can't, not even in theory. It's trivial to show that a Chinese Room that could converse for more than 15 minutes would have to contain more elementary particles than the entire planet.
There is an analogy here with Chomsky's linguistics. You can disagree with the specifics of his theory, but he did show that there was no way that people were learning sentences like parrots and repeating them. There had to be some computational / grammatical process.
Anybody can imagine mechanical alternatives for doing any information processing. To an extent, that drives the whole of the programming industry. The whole question of intelligence is that we can do it with a few pounds of biological material. Anything that doesn't address this is missing the point.
I don't believe it, can you show it?
So you think it's very obvious that it's impossible to build a chatbot that speaks chinese for 15 minutes in a computer like the one I'm using now (even one that claims to be a child for example). And why couldn't we just clone a Chinese person brain and say it's a computer? Why do you think a brain snapshot would require the size of a planet?
Ah, sorry, the "more elementary particles than the entire planet" claim wasn't made by me. My first comment was too vague, here is another try:
Yes, it sounds like a very strange claim, since isn't biological material made from elementary particles, and clearly you could fit someone who speaks Chinese into the room. Obviously that assumes that people are made of elementary particles, but if someone is going to argue that, I'm going to need a better definition of elementary particles to make any progress.
I thought Searle was asserting that a human mind can be simulated by a dumb program that only provided cached responses. It's easy to show there are more than 10e50 possible conversational paths in a 15 minute conversation (even a chess game has that many, and conversation is more complex than chess), and there are only 10e50 atoms on earth.
But it seems Searle's real argument was that the man inside the box is performing all the actions of an artificially intelligent computer program, just with pencil and paper. Humans clearly perform such actions with just a few pounds of brain matter. Even if the Chinese Room operator requires continent-wide resources, such difficulties don't address the essence of Searle's argument.
He seems to be asserting there's something special about the brain called "intentionality" which is unlike a computer running an algorithm. He might be right, but there seems to be no evidence for such a thing, or any way to prove that an entity which claimed "intentionality" didn't really have it.
Searle's thought experiment does inspire one of my own. Consider a computer that executes a sorting algorithm(mergesort, for example) on inputs consisting of sequences of number. Does it understand how to sort a sequence of numbers? What if a person uses mergesort manually to sort sequences of numbers? Does this person understand how to sort a sequence of numbers? Is there any fundamental difference between the understanding of a person who uses mergesort to sort sequences of numbers and that of a computer that uses mergesort to sort sequences of numbers?
It's a perfect trap for your intuition. You have an intuitive feeling that if you were to have a conversation in Chinese by following mechanical instructions from a book, you still wouldn't "really understand" Chinese. All well and good, so far.
But there are enormous leaps from the Searle-scenario that's so appealing to your intuition to a machine powerful enough to hold a conversation. As a trivial example, the Chinese room cannot even simulate a Turing machine unless you give the person inside rather a lot of scratch paper.
You can keep countering objections like this by adding more stuff to the Room. So to hold a conversation about the weather, we'll need to add some separate input channels to observe the outside world, and to talk about the Room's plans for today, we'll need to throw in something that can keep track of time, .... it just goes on. (And while you're adding all the extra machinery, don't forget that even the original book is incalculably large, not the merely hefty tome that you're picturing in your mind's eye.)
As you do this, you get farther and farther away from the original appeal to your intuition. So far that your only hope of making sense is to acknowledge that your intuitions about the simplified situation has absolutely nothing to do with a Chinese Room powerful enough to actually meet the original stipulation, that the room can "pass the Turing Test for understanding Chinese".
Right, this discussion is imcomplete without bringing in Church's thesis.
* So to hold a conversation about the weather, we'll need to add some separate input channels to observe the outside world, and to talk about the Room's plans for today, we'll need to throw in something that can keep track of time, .... it just goes on.*
Or these could be predefined and static.
(And while you're adding all the extra machinery, don't forget that even the original book is incalculably large, not the merely hefty tome that you're picturing in your mind's eye.)
This is like objecting that a Turing machine would take too many steps to solve a sufficiently large instance of some problem (e.g. more than there are Planck times in the age of the universe). The problem is still Turing-computable.
In what way is a hypothetical person using "a book of instructions for manipulating the symbols (the program)" different from a universal Turing machine executing the same instructions? When Turing manually executed his computer chess program, how did this differ from having a computer execute it?
"It's a perfect trap for your intuition. You have an intuitive feeling that if you were to have a conversation in Chinese by following mechanical instructions from a book, you still wouldn't "really understand" Chinese."
I fail to understand how this portion of what you're saying disagrees at all with what I said. I don't really see the rest of what you say as disagreeing with what I said, but it is at least not identical to it.
http://www.dreamsongs.com/Searle.html
"Searle's argument is subtle in a way that seems to confuse intelligent readers. "
Well now, that says something about philosophy's relevance to science.
> Many people consider the concept of gender inseparable from its fleshly and biological origin and nature.
And many people don't. The argument is just philosophical, what's the big deal?
If you take the focus off the word, much of your prejudice disappears. You see the relationships, the structure, the observations, the logic, the nuance. You can see you used two distinct meanings of the word "intelligence". One is a set of expected reactions, the other is your consciousness. Now this gives rise to a logical question: Are these two things the same? The Chinese room shows that's not necessarily the case. However, might that feeling you call consciousness be a side-effect of a the particular type of Chinese room that's going on in your head? That's how you get interesting philosophy.
No. Or at least, not entirely, not in any significant way, and mostly just no.
If language were truly subjective, we should expect almost all attempts at communication to fail. If language were radically subjective, then two people speaking to each other in the same language would be for all intents and purposes like a speaker of (only) English trying to communicate to a speaker of (only) Chinese. It's manifestly the case that this isn't so. Beyond this, if language were truly subjective, we wouldn't be able to think (since the person-to-person communication problem would emerge even for one person).
The real world strikes again.
Also, philosophers should try to realize that we have this thing called science now.