Rather superficial and obfuscating. The article keeps raising the question "why ignore the magnitude" and never answers it. "The important part of an embedding is its direction, not its length. If two embeddings are…
How big is the state space of the Markov chain corresponding to a LLM generating a sequence of tokens? Wouldn't it be (size of the vocabulary)^(size of the context window), i.e. ~ (100k)^(4k)? How useful is it to…
Mapping corresponding text and image into the same vector space is exactly how CLIP and other contrastive learning setups work: Have text and computer vision networks embed input data, and teach the networks to embed…
Rather superficial and obfuscating. The article keeps raising the question "why ignore the magnitude" and never answers it. "The important part of an embedding is its direction, not its length. If two embeddings are…
How big is the state space of the Markov chain corresponding to a LLM generating a sequence of tokens? Wouldn't it be (size of the vocabulary)^(size of the context window), i.e. ~ (100k)^(4k)? How useful is it to…
Mapping corresponding text and image into the same vector space is exactly how CLIP and other contrastive learning setups work: Have text and computer vision networks embed input data, and teach the networks to embed…