I'm not sure if I'm understanding correctly, but it reminds me of the kernel trick. The distances between the training samples and a target sample are computed, the distances are scaled through a kernel function, and the scaled distances are used as features.
He boiled all dimensionality reduction down to two approaches: matrix decomposition and k-nearest neighbors.
I'm seeing the newest contenders for top algorithms leaning harder into the KNN dynamics, and less into the more complex maths of UMAP. (PaCMAP[1] and LocalMAP[2], which run via an attractive-repulsive force simulation based on high dimensional KNN graph)
Based on the end of his talk, Leland seems to generally align that this is where the next innovations are: KNNs.
Some interesting provocations are coming out:
Assessing and improving reliability of neighbor embedding methods: a map-continuity perspective (Apr 2025)
https://arxiv.org/html/2410.16608v2
EDIT: that last paper is not as relevant as I recall on reviewing it further. So the point is mainly the take-away from Leland's expertise in his video
3 comments
[ 2.6 ms ] story [ 13.4 ms ] threadhttps://en.wikipedia.org/wiki/Kernel_method
Rethinking Unsupervised Learning https://youtu.be/v_jcu7rUqqE
He boiled all dimensionality reduction down to two approaches: matrix decomposition and k-nearest neighbors.
I'm seeing the newest contenders for top algorithms leaning harder into the KNN dynamics, and less into the more complex maths of UMAP. (PaCMAP[1] and LocalMAP[2], which run via an attractive-repulsive force simulation based on high dimensional KNN graph)
Based on the end of his talk, Leland seems to generally align that this is where the next innovations are: KNNs.
Some interesting provocations are coming out:
Assessing and improving reliability of neighbor embedding methods: a map-continuity perspective (Apr 2025) https://arxiv.org/html/2410.16608v2
[1]: https://youtu.be/sD-uDZ8zXkc
[2]: https://youtu.be/vmSUQ-4LT4Y
EDIT: that last paper is not as relevant as I recall on reviewing it further. So the point is mainly the take-away from Leland's expertise in his video