HN Explorer

Ask HN: We need a new Turing-Test

5 points by air7 ↗ HN
It wasn't long ago that the best attempt to pass the Turing-Test was a chatbot that "was portrayed as a 13-year-old Ukrainian boy to induce forgiveness in those with whom it interacts for its grammatical errors and lack of general knowledge." [0]. But then ChatGPT came along and blew the test out of the water.

There are other, more well defined intelligence tests such as the "Winograd Schema" [1] which asks grammatical questions that require real-world knowledge and common-sense in order to answer such as:

"In the sentence 'The trophy would not fit in the suitcase because it was too big/small.' what does big/small refer to?"

But even these type of questions which were considered to be "The Sentences Computers Can't Understand, But Humans Can" as late as 2020 [2] seem to be easy for LLMs to answer with GPT4 topping the list with almost human level accuracy [3].

So assuming ChatGPT isn't an AGI (yet), how can we prove it to ourselves? What linguistic tasks are humans qualitatively better than LLMs in?

------

[0] https://en.wikipedia.org/wiki/Eugene_Goostman [1] https://en.wikipedia.org/wiki/Winograd_schema_challenge [2] https://www.youtube.com/watch?v=m3vIEKWrP9Q [3] https://paperswithcode.com/sota/common-sense-reasoning-on-winogrande

4 comments

[ 4.0 ms ] story [ 20.0 ms ] thread
MattGaiser
Does ChatGPT really blow away the Turing test?

The criteria listed on Wikipedia is:

> The evaluator would be aware that one of the two partners in conversation was a machine

> If the evaluator could not reliably tell the machine from the human, the machine would be said to have passed the test.

ChatGPT isn't there by a long shot. If you aren't really considering that an online agent may be a bot and are just casually interacting with it, then it might pass. But within 10 minutes, it is not going to become anywhere near as friendly as a person normally would in 10 minutes of conversation, especially if you know that one is a bot and are looking for differences.

Turing Test is not casually passing as human in casual interaction. It is stumping a human dedicated to rooting out that it is a machine who is aware that one of the participants is a machine.

trealira
> Turing Test is not casually passing as human in casual interaction. It is stumping a human dedicated to rooting out that it is a machine who is aware that one of the participants is a machine.

It's a good point to bring up that what counts as "passing as human" is ambiguous, because people are using different definitions.

I'm reminded of how trans people say that passing is a spectrum, and not a binary. There's a difference between passing in the distance, passing from 20 meters away, passing when interacting with service workers (like cashiers, receptionists, etc.) or in other quick interactions, and passing with your friends, family, and coworkers. Maybe someone trans passes in quick interactions, but not in an extended face-to-face conversation with a stranger. I've also heard that people relatively ignorant of trans people are less likely to perceive them as their birth sex compared to people who are liberal or who aren't but spend a lot of time reading about them anyway (e.g. certain feminists and conservatives). Those people are likely to see someone wear a dress and take the mental shortcut of categorizing them as a woman, not paying attention to details that others would have, unless it's extremely obvious that they don't pass.

The same can be applied to AI. A language model might pass as a human to someone who doesn't know about AI; or to someone who knows about AI, but isn't expecting it in a certain context; but still not manage to stump the person who suspects and tries to figure out who's an AI and who's a human.

fortysixed
That's an interesting comparison because in both cases the intent is to deceive, and no matter how successful the deception is, the deceiver is still not what they purport to be. Hopefully future law and policy regarding AI will be crafted with that in mind, unlike that other set of deceivers who have had their deception enacted into law as if it is reality, in many jurisdictions.
Quinzel
Is it a fair test, if once it’s been passed, you change the testing criteria? I’m genuinely asking because I would have thought that the test was a measure of something (in this case whether AI is capable of thinking like a human being) , so then to change what you’re measuring because something actually passed the test seems to defeat the point of the test somewhat. Unless you want to redefine the variable being measured rather than the test? I guess “thinking like a human being” is a latent variable and not easily observable so changing the way you define it is possible, but then you could just forever be redefining what it means to think like a human and constantly ensure that AI never quite makes the cut?

That’s my philosophical pondering over the issue…