So did the previous generation models test higher when released, and now don't test as high? I may have misread the article but it seemed like the older models had a worse score all along, which can't really be called 'decline'?
...by which they mean "earlier-generation models have lower scores in a test measuring cognitive function than newer models".
The entire study was an exercise in academic clickbait, something that even other scientists complained about:
> Other scientists have been left unconvinced about the study and its findings, going so far as to critisize the methods and the framing — in which the study's authors are accused of anthropomorphizing AI by projecting human conditions onto it. There is also criticism of the use of MoCA. This was a test examined purely for use in humans, it is suggested, and would not render meaningful results if applied to other forms of intelligence.
The study authors defend with the classic "It's just a joke, bro" card:
> Responding to the discussion, lead author of the study Roy Dayan, a doctor of medicine at the Hadassah Medica Center in Jerusalem, commented that many of the responses to the study have taken the framing too literally. Because the study was published in the Christmas edition of the BMJ, they used humor to present the findings of the study — including the pun "Age Against the Machine" — but intended the study to be considered seriously.
The joke is that people tend to anthropomorphize LLM based chatbots so saying older models get age related diseases like humans is funny. The findings are the inverse, newer models do better on dementia tests than older models.
Whether a test was "intended" for humans or not is irrelevant in bench-marking AI. An AGI should reasonably be expected to answer such question.
5 comments
[ 2.5 ms ] story [ 26.5 ms ] threadThe entire study was an exercise in academic clickbait, something that even other scientists complained about:
> Other scientists have been left unconvinced about the study and its findings, going so far as to critisize the methods and the framing — in which the study's authors are accused of anthropomorphizing AI by projecting human conditions onto it. There is also criticism of the use of MoCA. This was a test examined purely for use in humans, it is suggested, and would not render meaningful results if applied to other forms of intelligence.
The study authors defend with the classic "It's just a joke, bro" card:
> Responding to the discussion, lead author of the study Roy Dayan, a doctor of medicine at the Hadassah Medica Center in Jerusalem, commented that many of the responses to the study have taken the framing too literally. Because the study was published in the Christmas edition of the BMJ, they used humor to present the findings of the study — including the pun "Age Against the Machine" — but intended the study to be considered seriously.
Whether a test was "intended" for humans or not is irrelevant in bench-marking AI. An AGI should reasonably be expected to answer such question.