Yes, but thats also what happens in the normal formulation, no? So the second weight matrix actually are our context embeddings?
My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the…
Yes, but thats also what happens in the normal formulation, no? So the second weight matrix actually are our context embeddings?
My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the…