In a nutshell, in Ayumi's "IQ" metric, the LLM ingests character cards intended for LLM roleplay, assumes the role of the character, and answers a series of multiple choice questions about themselves, using llama.cpp's custom grammar implementation to restrict the answers to 1,2,3,4, or 5. Its intended to test how well a model gets "in character" for roleplaying and storytelling.
More info on the metric itself and older tests is here, though the page is not up-to-date yet with the new testing:
This is super cool! I've grown distrustful of standard LLM ranking metrics, even when I'm pretty sure the testers aren't outright cheating, but this seems like a really novel test. What's more, no one is trying to game this RP/ERP testing, and the current ranking aligns with some of my own testing for LLM "smartness," even in models with no focus on roleplaying at all.
1 comment
[ 0.23 ms ] story [ 15.0 ms ] threadMore info on the metric itself and older tests is here, though the page is not up-to-date yet with the new testing:
https://rentry.co/ayumi_erp_rating#ayumi-llm-character-iq-al...
This is super cool! I've grown distrustful of standard LLM ranking metrics, even when I'm pretty sure the testers aren't outright cheating, but this seems like a really novel test. What's more, no one is trying to game this RP/ERP testing, and the current ranking aligns with some of my own testing for LLM "smartness," even in models with no focus on roleplaying at all.