So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token. lm_head just picks the closest thing and the model doesn't realize until too late.
Explains why RL helps. Base models never see their own outputs so they can't learn "this concept exists but I can't actually say it."
> So it's not really hallucinating - it correctly represents "seahorse emoji" internally, but that concept has no corresponding token.
I wonder if the human brain (and specifically the striated neocortical parts, which do seemingly work kind of like a feed-forward NN) also runs into this problem when attempting to process concepts to form speech.
Presumably, since we don't observe people saying "near but actually totally incorrect" words in practice, that means that we humans may have some kind of filter in our concept-to-mental-utterance transformation path that LLMs don't. Sometihng that can say "yes, layer N, I know you think the output should be O; but when auto-encoding X back to layer N-1, layer N-1 doesn't think O' has anything to do with what it was trying to say when it gave you the input I — so that output is vetoed. Try again."
A question for anyone here who is multilingual, speaking at least one second language with full grammatical fluency but with holes in your vocabulary vs your native language: when you go to say something in your non-native language, and one of the word-concepts you want to evoke is one you have a word for in your native language, but have never learned the word for in the non-native language... do you ever feel like there is a "maybe word" for the idea in your non-native language "on the tip of your tongue", but that you can't quite bring to conscious awareness?
To confirm, I tried this in ChatGPT and it produced a flood of wrong answers and self corrections just like that, scrolling so quickly that I couldn't read it until it eventually stopped itself.
* The LLM has strong and deep rooted belief in its knowledge (that a seahorse emoji exist).
* It attempts to express that concept using language (including emojis) but the language is so poor and inaccurate at expressing the concept that as it speaks it keeps attempting to repair.
* It is trained to speak until it has achieved some threshold at correctly expressing itself so it just keeps babbling until the max token threshold triggers.
I always felt like tokenization is one of those double edged swords where it makes some stuff amazingly easier but gets tripped up on the weirdest bugs. The number of "r"s in "strawberry" being another well-known quirk.
You'll also notice the same thing happens for other non-existent emojis that sound like they should exist: dragonflies, lemurs, possums, blackberries - even Claude 4.5 will start off by saying "Yes!" and then correcting itself. It will immediately give the right answer for very specific things that you wouldn't expect to get their own emojis though.
I realized if someone were to assign me the ticket for fixing this behavior, I would have no idea where to begin with solving it even with this blog post explaining the problem, so I'm very curious to know what the most practical solution is. (They obviously aren't adding "If someone asks you about a seahorse emoji, there isn't one available yet, no matter how strongly you believe one exists." to the system prompt.)
One explanation could be: many humans (including me) mistakenly think a seahorse emoji exists. My mind can even construct a picture of how it should look like, despite me also knowing it's very unlikely I've seen one myself.
A related questions: How do LLMs format code so consistently? I mean, when you write middle-indented things like struct fields in Go, how do they know in advance what the largest field name will be?
Fascinating. Gemini 2.5 Pro for me says that many believe it exists but it's actually an example of the Mandela effect. But ChatGPT 5.0 does do the whole thing and Claude does it for a bit before concluding it isn't real.
The generated text reminds me of Golden Gate Claude.
Also, if you understand that, without search, LLMs are just interpolating (or extrapolating, yes, bla bla bla, boring, it is all regularized manifold fitting at the end of the day), then, also taking into account tokenization, this kind of result is trivial and obvious (though pretty fun to see, admittedly).
This behavior reminds me a lot of what can happen to patients who have a corpus callosotomy.
In particular, one hemisphere will perform some action, and the other hemisphere will attempt to “explain” the behavior after the fact as if the intention was there all along.
I wonder to what degree the answer from Kagi Assistant is influenced by Kagi search or the underlying model. I asked Claude 4.5 Sonnet about the seahorse emoji because it was my default settings. The response was:
> The seahorse emoji does not officially exist in the Unicode emoji standard.
> Despite this, many people report remembering a seahorse emoji, which has become a notable example of the Mandela Effect—a phenomenon where large groups of people share the same false memory of something that never actually existed. [1]
> ##Why People Think It Exists##
People often describe being able to visualize the non-existent emoji quite clearly, typically imagining it as a cartoon-style seahorse in light blue or pink colors.
> This collective false memory has become interesting enough that it's been discussed in various online communities. [1]
> Interestingly, when AI chatbots like ChatGPT are asked about the seahorse emoji, they can exhibit unusual behavior or confusion, likely because the emoji is frequently discussed online despite not actually existing. [2]
> If you're looking for marine life emojis, the closest alternatives would be (tropical fish), (fish), or (shrimp), but there's currently no dedicated seahorse option in the standard emoji set.
113 comments
[ 1.2 ms ] story [ 84.5 ms ] threadExplains why RL helps. Base models never see their own outputs so they can't learn "this concept exists but I can't actually say it."
I wonder if the human brain (and specifically the striated neocortical parts, which do seemingly work kind of like a feed-forward NN) also runs into this problem when attempting to process concepts to form speech.
Presumably, since we don't observe people saying "near but actually totally incorrect" words in practice, that means that we humans may have some kind of filter in our concept-to-mental-utterance transformation path that LLMs don't. Sometihng that can say "yes, layer N, I know you think the output should be O; but when auto-encoding X back to layer N-1, layer N-1 doesn't think O' has anything to do with what it was trying to say when it gave you the input I — so that output is vetoed. Try again."
A question for anyone here who is multilingual, speaking at least one second language with full grammatical fluency but with holes in your vocabulary vs your native language: when you go to say something in your non-native language, and one of the word-concepts you want to evoke is one you have a word for in your native language, but have never learned the word for in the non-native language... do you ever feel like there is a "maybe word" for the idea in your non-native language "on the tip of your tongue", but that you can't quite bring to conscious awareness?
* The LLM has strong and deep rooted belief in its knowledge (that a seahorse emoji exist).
* It attempts to express that concept using language (including emojis) but the language is so poor and inaccurate at expressing the concept that as it speaks it keeps attempting to repair.
* It is trained to speak until it has achieved some threshold at correctly expressing itself so it just keeps babbling until the max token threshold triggers.
https://g.co/gemini/share/c244e5f51e37
And those text got into the training set: https://www.reddit.com/r/MandelaEffect/comments/qbvbrm/anyon...
https://www.gnod.com/search/ai#q=Is+there+a+seahorse+emoji%3...
Mistral being among the funniest ones:
Grok drew a frog for me and was convinced that it is a seahorse.“Yes, there is a seahorse emoji! It looks like this: <horse emoji><fish emoji> (sometimes rendered as a single emoji, depending on the platform).”
The generated text reminds me of Golden Gate Claude.
> No. There is no official seahorse emoji in Unicode. Closest: [gives list of closest emoji].
> source: https://emojipedia.org/seahorse-emoji-mandela-effect
Just enable search.
Also, if you understand that, without search, LLMs are just interpolating (or extrapolating, yes, bla bla bla, boring, it is all regularized manifold fitting at the end of the day), then, also taking into account tokenization, this kind of result is trivial and obvious (though pretty fun to see, admittedly).
In particular, one hemisphere will perform some action, and the other hemisphere will attempt to “explain” the behavior after the fact as if the intention was there all along.
> The seahorse emoji does not officially exist in the Unicode emoji standard.
> Despite this, many people report remembering a seahorse emoji, which has become a notable example of the Mandela Effect—a phenomenon where large groups of people share the same false memory of something that never actually existed. [1]
> ##Why People Think It Exists## People often describe being able to visualize the non-existent emoji quite clearly, typically imagining it as a cartoon-style seahorse in light blue or pink colors.
> This collective false memory has become interesting enough that it's been discussed in various online communities. [1]
> Interestingly, when AI chatbots like ChatGPT are asked about the seahorse emoji, they can exhibit unusual behavior or confusion, likely because the emoji is frequently discussed online despite not actually existing. [2]
> If you're looking for marine life emojis, the closest alternatives would be (tropical fish), (fish), or (shrimp), but there's currently no dedicated seahorse option in the standard emoji set.
1> https://www.reddit.com/r/MandelaEffect/comments/qbvbrm/anyon... 2> https://futurism.com/chatgpt-haywire-seahorse-emoji