Why do LLMs fail to count letters in a word? (anands.me)

I was thinking that LLMs could benefit from knowledge about their tokenization, particularly if you are using them for LLM development.

Even humans cannot do this task if it's a long word unless we "spell" or "look" at individual letters and memorise the count.

Attention and tokenization seems to be the problem with LLMs

No letters, only words