Evaluating GPT5's reasoning ability using the Only Connect game show (ingram.tech)

It's extremely interesting. I do have a suggestion to make sure the question are not in training data:

If this game show is like the ones in my country, people do come together in 'clubs' to train for the event, sometimes organising internal tournaments. Some people in those clubs are question-writers, and write the internal tournaments questions.

Maybe try to contact those clubs, and find those writers, you'll be sure the LLM won't be trained on that specific set.

I cannot, cannot, cannot recommend Only Connect enough. Readers here should enjoy it. As an American, a lot of UK geography and history questions are beyond my ken, but it's well-worth your time. Victoria Coren Mitchell is a national treasure.

It's sort of exactly what I'd think LLMs would be good at. Maybe not so much the lateral thinking—the occasional "math(s) question but actually it's a word one"—but rapidly assessing "what connects these" or "what would continue this sequence" is right up their alley. The joy, of course, is trying it yourself and seeing others do the same.

That's great, I thought about building a similar thing data from PuzzGrid, an onlyconnect fan site, but some of the questions there are a bit iffy compared to the ones on the show. How did you build the dataset - just binge watching with a notepad?

As someone who loves the show this is very fun to see. I'm impressed at the number of correct answers most get.

Helpfully as well there is a current season on, and while the total questions wouldn't be enough to fully validate results while ensuring the data isn't in the training it's certainly good enough as a sanity check that people could do.

> Wall: Players group 16 elements into four categories (similar to the NYT Connections game)

I have to be the designated pedant here and point out that Only Connect was first.

I tried for a while to get ChatGPT to generate connections style puzzles with some suggested topics, including red herrings to create some answers that seemingly fit in multiple categories. Then it would post them to https://connections.swellgarfo.com/. Overall they were really bad but that was using GPT4

Good job being explicit about the reasoning effort and verbosity settings. And interesting but not surprising that verbosity helped performance.

So basically - LLM usefulness has plateau’d hard - Sam A knows this and will pivot to capturing revenue now that the high growth phase is over - more restrictive rate limits, higher bills

They U-turned on the recent rate limit changes after the release backlash but more is coming

Anecdotally it doesn't feel like GPT-5 is a meaningful improvement over o3 to me.

>We were unable to find evidence that the Only Connect games are in the training materials (which of course is likely to change now).

I just don't think this is a credible assumption. The BBC is one of the highest-trusted sources of millions of hours online audio/visual content, all of which is accompanied by human curated & edited closed captions. All of which is trivially easy to download. The base assumptions should be that the entire BBC iPlayer corpus is inside of all the frontier model training datasets.

The communities on Reddit (known to be included in all models) extensively discuss each show & question - usually creating google docs tracking the questions asked and answers given.

Finally, there's the OCDB[0], which lists every question and answer on the show.

While using real questions from the show, this benchmark should be assumed to be testing the model fact-recall ability, rather than its reasoning capabilities.

[1]https://ocdb.cc/

Did you use o3 pro high, o3 high or o3 medium?

(OpenAI's naming is so confusing)

> We were unable to find evidence that the Only Connect games are in the training materials (which of course is likely to change now).

Respectfully, I do not think this is a good assumption for any TV show broadcast prior to 2025.

e: tonality wise, llm threads seem to bring out the worst

The top reasoning competitor to GPT-5 would probably be Gemini 2.5 Pro. That is the first model I would like to see it compared with.

I am less interested in questioning training data corruption than I am in questioning claims like this:

  test reasoning abilities such as pattern recognition, lateral thinking, abstraction, contextual reasoning (accounting for British cultural references), and multi-step inference.... its emphasis on clever reasoning rather than knowledge recall, Only Connect provides an ideal challenge for benchmarking LLMs' reasoning capabilities.

It seems to me that the null hypothesis should be "LLMs are probabilistic next-word generators and might be able to solve a lot of this stuff with shallow surface statistics built from inhumanly large datasets, without ever properly using abstraction, contextual reasoning, etc." This is particularly true for NYT Connections, but in general evaluations like this seem to be at least partially testing how amenable certain word/trivia games are to naive statistical algorithms. (Many NYT Connections "purple" categories seem like they would be quite obvious to a next n-gram calculator, but not for people who actually use words conversationally!) Humans don't use these statistical algorithms for reasoning except in particular circumstances (many use "folk n-gram statistics" when playing Wordle; poker; serious word game players often learn more detailed tables of info; you could see competitive NYT Connections players learning a giant bag of statistical heuristics to help them speedrun things). We just can't accumulate the data ourselves without making a concerted computer-aided effort.

In general a lot of LLM benchmarks don't adequately consider that LLMs can solve certain things better than humans without using reasoning or knowledge. The most stupid example is how common multiple choice benchmarks are, despite us all learning as children that multiple-choice questions can be partially gamed with shallow statistical-linguistic tricks even if you have no clue how to answer the question honestly[1]; it stands to reason that a superhuman statistical-linguistic computer could accumulate superhuman statistical-linguistic tricks without ever properly learning the subject matter. AI folks have always been quick to say "if it quacks like a duck it reasons like a duck" but these days computers are quite good at playing duck recordings.

[1] "When in doubt, C your way out," sniffing out suspicious answers, shallow pattern-matching to answer reading comprehension, etc etc. One thing humans and LLMs actually do have in common is that multiple-choice tests are terrible ways to assess their knowledge or intelligence.