I only recently started learning about vector embeddings, and the thing I can't wrap my head around is why cosine similarity should be so effective when (to my understanding) it should only capture linear similarity.
Shouldn't that "linear" constraint have more of a blunting effect on embeddings effectiveness? Why not use a different metric like mutual information that can account for non-linearities. Whenever correlation comes up in context of statistical comparison, there's always this warning that correlation is limited as a way of comparing data due to it's linearity so I'm confused why it seems so effective in the context of embeddings.
Mutual information relies on probability distributions, so I don’t think it makes sense to compare two random embeddings using it. Also, cosine similarities aren’t used in training BERT, I think it’s just cross entropy loss, which makes sense when doing classification.
Very good point. This is an active area of research, there's been some work in using other similarity measures like MI,Kendall's Tau etc in embeddings as well. And they have been found to perform pretty well.
EDIT: As @rubatuga said,they have been avoided largely as embeddings are continuous random vars and MI is for distributions. Nonetheless I think there can be merit in exploring these
PS: I am the author and just wanted to talk about the familiar basic stuff on this one
The encoder is a neural network which is a nonlinear and highly expressive function family. The encoder function is optimized to produce a set of "factors" which are designed to be linearly comparable. If the "factors" aren't expressive enough, you can add more (increase the embedding dimension) or make them more sophisticated (increase the encoder network complexity). There's nothing stopping you from allowing them to interact in a more complex way as well, except that you give up the ability to easily do operations like indexing.
You've hit the nail on the head: the early Word2Vec models treat everything as a word, but words are polysemous and fluid. Mutual information is useful for discovering n-grams that can be informatively mapped to the same vector space as component 1-grams. (In fact, converting adjacent words with high pointwise mutual information such as "New" and "Zealand" to "New_Zealand" is an important preprocessing step noted by Mikolov, et al).
More recent vector embeddings (e.g., BERT, ELMO) are harder to conceptualize neatly because one of the layers of the Neural Network includes a token's indexical position in the document.
One of the most famous word embedding methods, Word2Vec, actually approximates word co-occurrence PMI matrix through matrix decomposition. Check out Omer Levy and Yoav Goldberg's work.
In that case given enough dimensions it could approximate any matrix well - i.e. linearity shouldn't be a limiting factor.
Correlation (vs causation) and linearity are orthogonal concepts.
for all common models (GloVe, fastText, word2vec) the means across word embeddings are tightly concentrated around zero (relative to their dimensions), thus making the widely used cosine similarity practically equivalent to Pearson correlation https://www.aclweb.org/anthology/N19-1100/
It may be misleading to treat noncontextual word embeddings (e.g. GloVe, word2vec) and contextual word embeddings (BERT, ELMO, etc) as equivalent; there's a substantial contextual difference whether you're talking about a vector representing "boy" as a generic string or concept, equal for all mentions of "boy", or if you're talking about a vector representing a specific mention of "boy" in a specific context only, with no grounds to assume that it will be equal or even similar to other mentions of "boy". The former embeddings [try to] capture generic meaning of words, the latter [try to] capture the more specific meaning of particular utterances.
9 comments
[ 2.8 ms ] story [ 27.0 ms ] threadShouldn't that "linear" constraint have more of a blunting effect on embeddings effectiveness? Why not use a different metric like mutual information that can account for non-linearities. Whenever correlation comes up in context of statistical comparison, there's always this warning that correlation is limited as a way of comparing data due to it's linearity so I'm confused why it seems so effective in the context of embeddings.
I found something for MI if you wanna check it out : https://www.aclweb.org/anthology/2020.acl-main.741.pdf
EDIT: As @rubatuga said,they have been avoided largely as embeddings are continuous random vars and MI is for distributions. Nonetheless I think there can be merit in exploring these
PS: I am the author and just wanted to talk about the familiar basic stuff on this one
More recent vector embeddings (e.g., BERT, ELMO) are harder to conceptualize neatly because one of the layers of the Neural Network includes a token's indexical position in the document.
Correlation (vs causation) and linearity are orthogonal concepts.