Pokémon Red is also becoming a go-to benchmark for testing the agentic abilities of advanced AI models. But is Pokémon Red actually a good eval for LLMs? We study this problem in a standardized setting and identify three big issues: 1⃣ Without scaffolding, even top models can’t play Pokémon Red. 2⃣ Eval setups differ across reports—no fair comparison. 3⃣ It’s expensive! Very low cost-effectiveness.

We share more details and findings in our blogpost: https://lmgame.org/#/blog/pokemon_red