Yeah one of the first open-source recommendation engines I ever worked with was called Voogo[1] and I believe it was based on k-means. This was back in 2008 or so?
For someone who had never been exposed to any of the math behind this kind of thing, it was an interesting implementation, and the source code was very readable.
The original website seems to be gone and I couldn't find a Git link so apologies for Sourceforge.
Might have been k-nearest-neighbors rather than k-means. Knn can be used for "recommended because you bought X" or "users like you also bought X" type recommendations that relate user to user or item to item.
K-means could potentially be helpful to group together common users/items if e.g. you're memory constrained and don't want to give each user a fully unique embedding entry so that's also possible.
I'm no expert in ML, but beyond the research and emerging work in unsupervised learning clustering seems to be the most common approach, and there's nothing conceptually new here in the past 10+ years. Don't get me wrong, computationally we can do stuff with a ridiculous # of dimensions that was hard/impossible before and there are new algorithms but I was doing KMeans and DBScan/HDBscan and Gaussian mixtures in grad school 15 years ago in relation to databases, I had never heard of "Machine Learning" and we were in the glacial stage of the AI winter. There's some newer work that is based on human judgement for results but clustering still seems to be the mainstream "data validated" approach...
I have a data set from the 1980s and the programs to apply these techniques to it, written in compiled BASIC (predates Microsoft QuickBASIC). They ran their analyses on run of the mill IBM PCs. Really wish I could get it all into a modern framework…
The authors published a respected book based on the data and used it as the foundation for a bunch of other applied research. Sometimes I wish I could resurrect those researchers, or at least have one of their ghosts stop by and see what we can do with computers today.
it's still widely used and can be validated without significant human judgement. Implementations are more efficient, and computatal complexity is through the roof but the approach is still legit.
What would the GLM predict as an outcome? UMAP is fine for dimension reduction, but it doesn't output classes. Clustering, prediction (class or value), and dimensionality reduction are all trying to get at different aims with at best a little bit of overlap.
I would say to look at credit utilization as a function of the other variables. Or more generally each variable as a function of the other variables. There is a bit of this in the initial correlation analysis, but a GLM is more robust.
I'm gonna say his analysis of the groups is just wild guessing.
> Mostly married with low credit limits and high utilization rates. These customers may benefit from tools and programs to manage their credit usage.
> Mostly single women with low c redit limits and high utilization rates. Offering credit management tools and financial literacy programs could be valuable for this group.
K-means is like the McDonald's of clustering algorithms. It's fast, it's convenient, and sometimes you use it because it's fast and it's convenient...but if you care about quality, it's seldom the right choice. Assuming all your clusters are spherical Gaussians may be convenient but is very often unrealistic.
Depends on your data and what assumptions you can make. I work with sequence data a lot, and for that type of data the MMSeqs library (https://github.com/soedinglab/MMseqs2) is both very powerful and very popular.
For tabular data as in this blog post, there are a lot of options. For small datasets, hierarchical clustering is very powerful -- you can build and visually inspect a dendrogram and this can give you a lot of insight. It's implemented in Scipy and scikit-learn (e.g. https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy... ). Hierarchical clustering however scales poorly. For relatively low-dimensional data the hdbscan algorithm is really nice and is implemented in Python (https://pypi.org/project/hdbscan/).
If you have reason to think your data is modeled reasonably well as mixture of Gaussians (think lots of elliptical clusters of various sizes) and it's not too high-dimensional, a mixture of Gaussians can work well; unlike k-means, it's probabilistic and doesn't assume all clusters are spherical and roughly equal in size. This too is implemented in scikit-learn (https://scikit-learn.org/stable/modules/generated/sklearn.mi...). If you think a mixture of Gaussians is reasonable but you know there are outliers, a mixture of Student t-distributions will work better; this is not in scikit-learn but there are multiple implementations on github.
It's also possible to improve k-means by using approximate kernel k-means, where you use a random Fourier features (https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.n...) representation for each input datapoint then run k-means on that -- this approximates kernel k-means, so it relaxes some of the unrealistic assumptions of k-means. We no longer assume clusters are spherical, although this method may still work poorly if there are outliers and also still requires us to choose the number of clusters and the lengthscale for the kernel we are approximating, both of which may be hard to choose unless you already have a pretty good intuition for what are appropriate choices for your data.
There are other options that are sometimes useful, in fact, I could easily write a blog post about this (maybe I should), but the thing with clustering is that the "right" choice of algorithm is somewhat dependent on your data and on what assumptions are reasonable to make. People sometimes end up using k-means because it's fast (especially if you use minibatch kmeans) and can scale to crazy large datasets. But it makes very strong assumptions which are usually wrong (most datasets do not subdivide well into some number of spherical Gaussians of roughly equal size), and this can result in truly absurd partitions, especially when there are lots of outliers or clusters with highly irregular shapes.
It depends on your goals. In something like bolt (a vector quantization algorithm), it's provably the right choice (and with other preferred error measures for your quantization, analogous measures on kmeans are still the right choice).
29 comments
[ 13.2 ms ] story [ 80.7 ms ] threadhttps://archive.is/9Yytc
The result is surprise-free.
[1] https://github.com/Medalytics/Dataquest-Projects/blob/main/P...
Artisanal AI.
For someone who had never been exposed to any of the math behind this kind of thing, it was an interesting implementation, and the source code was very readable.
The original website seems to be gone and I couldn't find a Git link so apologies for Sourceforge.
1: https://sourceforge.net/projects/vogoo/
K-means could potentially be helpful to group together common users/items if e.g. you're memory constrained and don't want to give each user a fully unique embedding entry so that's also possible.
Thanks for the correction
And by the way, neural networks aren’t new either.
The authors published a respected book based on the data and used it as the foundation for a bunch of other applied research. Sometimes I wish I could resurrect those researchers, or at least have one of their ghosts stop by and see what we can do with computers today.
I'm looking for some self-paced learning on data science for developers.
This is incredibly sad. They are so distinct that a cluster Of these people is extracted. Absent fathers are a curse on society.
> Mostly married with low credit limits and high utilization rates. These customers may benefit from tools and programs to manage their credit usage.
> Mostly single women with low c redit limits and high utilization rates. Offering credit management tools and financial literacy programs could be valuable for this group.
For tabular data as in this blog post, there are a lot of options. For small datasets, hierarchical clustering is very powerful -- you can build and visually inspect a dendrogram and this can give you a lot of insight. It's implemented in Scipy and scikit-learn (e.g. https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy... ). Hierarchical clustering however scales poorly. For relatively low-dimensional data the hdbscan algorithm is really nice and is implemented in Python (https://pypi.org/project/hdbscan/).
If you have reason to think your data is modeled reasonably well as mixture of Gaussians (think lots of elliptical clusters of various sizes) and it's not too high-dimensional, a mixture of Gaussians can work well; unlike k-means, it's probabilistic and doesn't assume all clusters are spherical and roughly equal in size. This too is implemented in scikit-learn (https://scikit-learn.org/stable/modules/generated/sklearn.mi...). If you think a mixture of Gaussians is reasonable but you know there are outliers, a mixture of Student t-distributions will work better; this is not in scikit-learn but there are multiple implementations on github.
It's also possible to improve k-means by using approximate kernel k-means, where you use a random Fourier features (https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.n...) representation for each input datapoint then run k-means on that -- this approximates kernel k-means, so it relaxes some of the unrealistic assumptions of k-means. We no longer assume clusters are spherical, although this method may still work poorly if there are outliers and also still requires us to choose the number of clusters and the lengthscale for the kernel we are approximating, both of which may be hard to choose unless you already have a pretty good intuition for what are appropriate choices for your data.
There are other options that are sometimes useful, in fact, I could easily write a blog post about this (maybe I should), but the thing with clustering is that the "right" choice of algorithm is somewhat dependent on your data and on what assumptions are reasonable to make. People sometimes end up using k-means because it's fast (especially if you use minibatch kmeans) and can scale to crazy large datasets. But it makes very strong assumptions which are usually wrong (most datasets do not subdivide well into some number of spherical Gaussians of roughly equal size), and this can result in truly absurd partitions, especially when there are lots of outliers or clusters with highly irregular shapes.