Overcomplete nature of DNN is a common phenomenon and often advocated to convert saddle points into local minima --- understanding the actual landscape is difficult due to the inherent nonlinearity.... this indeed leads…
Just to clarify a bunch of questions below: The idea is more like : start from a full embedding layer train the network, after the training stabilizes break the first layer into two using the SVD formulation in an…
The paper is here: https://arxiv.org/pdf/1811.00641.pdf
The idea ismore like : start from a full embedding layer train the network, after the training stabilizes break the first layer into two using the SVD formulation in an online fashion and continue training - after only…
Overcomplete nature of DNN is a common phenomenon and often advocated to convert saddle points into local minima --- understanding the actual landscape is difficult due to the inherent nonlinearity.... this indeed leads…
Just to clarify a bunch of questions below: The idea is more like : start from a full embedding layer train the network, after the training stabilizes break the first layer into two using the SVD formulation in an…
The paper is here: https://arxiv.org/pdf/1811.00641.pdf
The idea ismore like : start from a full embedding layer train the network, after the training stabilizes break the first layer into two using the SVD formulation in an online fashion and continue training - after only…