Current LLMs go crazy trying to handle "Are there primes whose digits sum to 9?" (twitter.com)

I tested this myself in GPT4-o before submitting the link and immediately GPT4 told me that 3 is a valid answer because 3 = 9, among other wrong answers.

Trying to help it towards the correct "no" answer seems to only drive it further into confusion. It's like the strong innate desire to provide a helpful example drives it to insanity instead of just saying that it doesn't know any valid solutions.

it would be f* funny if in the future, AI will be wasting 90% of energy on trying to resolve the Riemann hypothesis secretly from the humanity.