Set is a card game where players have to identify sets of three cards from a layout of 12. Each card features a combination of four attributes: shape, color, number, and shading. A valid set consists of three cards where each attribute is either the same on all three cards or different on each. The goal is to find such sets quickly and accurately.
Though this game is a solved computer problem — easily tackled by algorithms or deep learning — I thought it would be interesting to see if Large Language Models (LLMs) could figure it out.
FYI: Card 8's transcription is different than the image. In the image 5, 8, 12 is a Set but the transcription says Card 8 only has 2 symbols which removes that Set.
Oh no, thanks for pointing this out! I asked GTP-4o to convert the image to text for me and I only checked some of the cards, assuming the rest would be correct. That was a mistake.
I've now corrected the experiment to accurately take the image into account. This meant that Deepseek was no longer able to find all the sets, but o3-mini still did a good job.
I believe GP is being sarcastic, as this seems to be a common reaction. Every time a machine accomplishes something that seems to be intelligent, we redefine intelligence to exclude that thing, and now the issue is fixed: computers are not intelligent and they cannot think.
Never mind that they can beat the entire world at chess and Go, and 90+% of the population at math, engineering, and physics problems. Those things do not require intelligence or thinking.
I still think that we’re at much greater risk of discovering that human thinking is much less magical, than we are of making a machine that does magical thinking.
Since you can train an LLM to play chess from scratch, I would not be surprised if you could also train one to play Set. I might experiment with it tomorrow.
I noticed that LLM at least at the Claude and OpenAI 4o level can not play tic tac toe and win against a competent opponent. They make illogical moves.
Interestingly, they can write a piece of code to solve Tic Tac Toe perfectly without breaking a sweat.
I've always said that appending "use python" to your prompt is a magic phrase that makes 4o amazingly powerful across a wide range of tasks. I have a whole slew of things in my memories that nudge it to use python when dealing with anything even remotely algorithmic, numeric, etc
33 comments
[ 4.5 ms ] story [ 71.8 ms ] threadThough this game is a solved computer problem — easily tackled by algorithms or deep learning — I thought it would be interesting to see if Large Language Models (LLMs) could figure it out.
I've now corrected the experiment to accurately take the image into account. This meant that Deepseek was no longer able to find all the sets, but o3-mini still did a good job.
Never mind that they can beat the entire world at chess and Go, and 90+% of the population at math, engineering, and physics problems. Those things do not require intelligence or thinking.
Tangentially related: https://qntm.org/mmacevedo
This is wildly disconcerting to me
https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
(this one, where ever changing rules is part of the game: https://www.looneylabs.com/games/fluxx )
Interestingly, they can write a piece of code to solve Tic Tac Toe perfectly without breaking a sweat.
On the other hand writing a piece of code to solve Tic Tac Toe sounds like it could be a relatively common coding challenge.