Token Embeddings Violate the Manifold Hypothesis

If this interests you, https://bsky.app/profile/jefferyharrell.bsky.social has been making some super cool visualizations of Qwen3's internal representations

One thing discovered was that 800+ tokens converged to a single point in the 2500+ dimensional unembedding space

Some examples:

Token Embeddings Violate the Manifold Hypothesis (arxiv.org)