The author never implied that this was their own original discovery. Unless they ripped the entire article off, this is just a tutorial on how to work with Eigenfaces on your own, and an explanation of how they work.
Nonetheless, I've found it to be an interesting concept. There is indeed a homework from the Coursera ML course for computing and visualizing eigenfaces, and the course (and the Stanford CS229 notes) discuss PCA further. I decided to explore the ideas further and distill it into a blog post specifically on eigenfaces.
Goal is for it to serve as a condensed tutorial on an interesting topic! I definitely learned a bunch writing it, and it may be interesting to others who have yet to come across to concept.
It's not completely by chance. There's an old paper [1] that shows that if the activation functions are well approximated using only up to the linear term of its Taylor expansion, then the optimal weights for encoding and decoding are the same as PCA.
There's probably newer results on this topic; I'm sure.
However, I will say that I've created some autoencoders on toy sets like those found in scikit-learn, and the spaces learned via the autoencoder and the spaces found through PCA were often similar if not identical. For example, if my input vectors were in R^n (with n > 3) and I restricted an autoencoder to 3 units, the encoding matrix of the autoencoder would span the same subspace as the first 3 principal component directions.
(Nomen omen) spectral look is an artifact of using negative values. It's nice to see non-negative components - http://www.quantumblah.org/?p=428. They are both more accurate and more human-interpretable (at the cost of computational efficiency).
15 comments
[ 2.7 ms ] story [ 21.9 ms ] threadhttp://jeremykun.com/2011/07/27/eigenfaces/
http://nbviewer.ipython.org/github/rcquan/sklearn-practice/b...
The wikipedia article (https://en.wikipedia.org/wiki/Eigenface) also contains code for a MATLAB implementation.
Nonetheless, I've found it to be an interesting concept. There is indeed a homework from the Coursera ML course for computing and visualizing eigenfaces, and the course (and the Stanford CS229 notes) discuss PCA further. I decided to explore the ideas further and distill it into a blog post specifically on eigenfaces.
Goal is for it to serve as a condensed tutorial on an interesting topic! I definitely learned a bunch writing it, and it may be interesting to others who have yet to come across to concept.
https://lambdal.com/images/autoencoder-learning-face-filters...
There's probably newer results on this topic; I'm sure.
However, I will say that I've created some autoencoders on toy sets like those found in scikit-learn, and the spaces learned via the autoencoder and the spaces found through PCA were often similar if not identical. For example, if my input vectors were in R^n (with n > 3) and I restricted an autoencoder to 3 units, the encoding matrix of the autoencoder would span the same subspace as the first 3 principal component directions.
[1]: http://oucsace.cs.ohiou.edu/~razvan/courses/dl6900/papers/bo...
(•_•) ( •_•)>⌐■-■ (⌐■_■)
http://www.mitpressjournals.org/doi/abs/10.1162/jocn.1991.3....
nearly 25 years old and 13k references, so it's pretty well studied...!
I really should add a section of resources that I found useful. Thanks!
One of the authors is still working on refining their approach, by the looks of it: http://gravis.cs.unibas.ch/projects.html
For those wondering why PCA works (self-plug): http://ilyakava.tumblr.com/post/95691347612/demystifying-pca