The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.
Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.
Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.
Eigenvectors answer the question, if you have a linear mapping from a space to itself: which lines through (through the origin) remain unchanged after the mapping? And the associated eigenvalues tell you how much a specific point on that line moves along the line.
For example: If you rotate things in 3D, the rotation axis remains unchanged. That's the eigenvector.
If you reflect things through a mirror plane, any line in that mirror plane will remain unchanged (with eigenvalue 1). Other lines will be, well, mirrored, but there is one line that is reflected onto itself: the one perpendicular to the mirror plane. However, points on it will go from one side to the other, so have eigenvalue -1.
The SVD says something else: Any linear mapping from a space (to itself or a different one - higher or lower dimensional) can be expressed as 1) first a rotation in the old space, then 2) some coordinate scaling or dilation (expansion/contraction) along the coordinate axes, described by the diagonal matrix with singular values, 3) then another rotation in the new space.
If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0].
I'm not claiming it's a practical way to do so, but this means that if you set up a neural network w/o nonlinearities that goes U -> S -> V^T, where S is a truncated scaling vector, and U and V^T are trained weights, make your loss function the MSE of reconstruction error, and minimize it with gradient descent, you will end up with the same U, S, and V that an SVD gives you.
In fact, this is basically exactly what a Variational Autoencoder [1] is! Way too few people realize this connection, and I wish it was taught in more ML courses. VAEs just add nonlinearities between U -> nonlinearity -> S -> nonlinearity -> V^T, and a KL-divergence regularization term. (Well VAEs are trained as operators to reconstruct vectors, and the S is an embedding not a trained weight, so I'm being a little sloppy, but still the connection is strong).
Once you realize this, you can have a lot of fun... anywhere you see an SVD being useful, you can construct arbitrary neural networks to replace them, and any time an SVD doesn't quite fit, e.g. you have binary data, realize that VAEs are just the same thing you can make all kinds of bespoke changes to... don't want MSE as your reconstruction error? Fine, use something else, but it's basically just an SVD!
BTW, if you wonder about the dedication ("For Gene Golub on his 15th birthday"):
Gene Golub was a numerical analyst, and father of the practical singular value decomposition (his license plate read "Prof SVD"), together with William Kahan (the father of IEEE 754 floating point numbers).
And his birthday was February 29. (In other words, the article was on the occasion of Gene's 60th birthday).
Just to add: Credit is also due to Prof Christian Reinsch from Technical University of Munich (in addition to Golub and Kahan) for the invention of the practical algorithm for computing the SVD still in use today.
9 comments
[ 3.3 ms ] story [ 27.6 ms ] threadSingular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.
Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.
Eigenvectors answer the question, if you have a linear mapping from a space to itself: which lines through (through the origin) remain unchanged after the mapping? And the associated eigenvalues tell you how much a specific point on that line moves along the line. For example: If you rotate things in 3D, the rotation axis remains unchanged. That's the eigenvector. If you reflect things through a mirror plane, any line in that mirror plane will remain unchanged (with eigenvalue 1). Other lines will be, well, mirrored, but there is one line that is reflected onto itself: the one perpendicular to the mirror plane. However, points on it will go from one side to the other, so have eigenvalue -1.
The SVD says something else: Any linear mapping from a space (to itself or a different one - higher or lower dimensional) can be expressed as 1) first a rotation in the old space, then 2) some coordinate scaling or dilation (expansion/contraction) along the coordinate axes, described by the diagonal matrix with singular values, 3) then another rotation in the new space.
If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0].
I'm not claiming it's a practical way to do so, but this means that if you set up a neural network w/o nonlinearities that goes U -> S -> V^T, where S is a truncated scaling vector, and U and V^T are trained weights, make your loss function the MSE of reconstruction error, and minimize it with gradient descent, you will end up with the same U, S, and V that an SVD gives you.
In fact, this is basically exactly what a Variational Autoencoder [1] is! Way too few people realize this connection, and I wish it was taught in more ML courses. VAEs just add nonlinearities between U -> nonlinearity -> S -> nonlinearity -> V^T, and a KL-divergence regularization term. (Well VAEs are trained as operators to reconstruct vectors, and the S is an embedding not a trained weight, so I'm being a little sloppy, but still the connection is strong).
Once you realize this, you can have a lot of fun... anywhere you see an SVD being useful, you can construct arbitrary neural networks to replace them, and any time an SVD doesn't quite fit, e.g. you have binary data, realize that VAEs are just the same thing you can make all kinds of bespoke changes to... don't want MSE as your reconstruction error? Fine, use something else, but it's basically just an SVD!
[0] https://en.wikipedia.org/wiki/Low-rank_approximation#Basic_l... [1] https://en.wikipedia.org/wiki/Variational_autoencoder
https://linear.axler.net/
Gene Golub was a numerical analyst, and father of the practical singular value decomposition (his license plate read "Prof SVD"), together with William Kahan (the father of IEEE 754 floating point numbers).
And his birthday was February 29. (In other words, the article was on the occasion of Gene's 60th birthday).
https://en.wikipedia.org/wiki/Gene_H._Golub
RIP.
https://blogs.mathworks.com/cleve/2022/10/23/christian-reins...
https://people.inf.ethz.ch/gander/talks/Vortrag2022.pdf
https://www.mathworks.com/company/technical-articles/profess...
https://en.wikipedia.org/wiki/Christian_Reinsch