My experience is that you can't have a generic good recommender system. It all depends on your business but if you just want a mixture between using navigation data and item data lightfm works pretty well for the generic case
I don't really mean generic. Heck, I'd be curious to know which one is the best in its most ideal use. That is, have any of them ever been really good?
I'm coming at this from the perspective that recommendation engines are, as far as I can tell, practically a hoax.
They aren't a hoax, some type of recommendation engines works really well especially the collaborative filtering types.
The reallity is that once you have a subset of items that matches your behavior recsys are learning to rank problems in the broad sense, and any kind of algorithmic feed (fb, Instagram, news, Twitter) is essentially a recsys.
There is a whole conference around these subjects with very good papers and competitions, look at the different kind of approaches to the recsys18 challenge by Spotify at the ACM site
But that gets us back to the "Theory is closer to practice in theory more than practice" problem.
some type of recommendation engines works really well especially the collaborative filtering types.
Exactly what I was asking about, because I'm not so sure. Furthermore, I think it's a category error to include collaborative filtering here, which is not automated/ML/etc.
What's HNs best sociological guess for why recommendation systems are still pretty crappy? It can't be that it's technically or mathematically too challenging to figure out. There are too many smart people out there for this not to be cracked and then immediately spread to everyone. I feel like there's a Peter Thiel-esque explanation to it. Something about a lack of diversity of thought and/or the lack of an incentive to encourage first-principles inventions in software engineering (or a lack of investments into long-term innovations which benefit everyone).
Because you're focusing on a technology solution. This isn't a technology problem. We cannot model the impossibly complicated measure of taste, the best we can do is optimize on captured data and generalize. This technique - using optimization functions over sparse matrices - has taken us as far as it can go.
I think recommendation systems actually are pretty good at recommending stuff you already like. I think a lot of us tend to buzz right past those as "well, obviously I've seen that already" and don't take note that it actually hit the nail right on the head for the first 5-10 results.
The trick is to find stuff that cuts across the space in some sense—stuff that's out of your current wheelhouse, but still has features that you would like. (E.g. I like the music of both David Rawlings and Mark Ribot, even though they are in wildly different genres, because they both have a certain ear for melody and relationship with dissonance that I like.) I don't think anyone is doing that well yet.
Collaborative systems are designed to do this. If enough people who like A also like B, and you like A but haven't rated B, they should predict you will like B.
One of the reasons, I suspect, is that there is a lack of feedback. If you recommend something to a friend, later on you can ask them about it and they'll tell you whether they liked it or not and why. Whereas these recommendation engines keep recommending things but have no way of knowing whether the recommendations they make are any good.
Wow, this is very perceptive and could possibly be mitigated. For instance, a lot of the recommendation engines I've looked at use basic actions (like clicks) to determine whether something is worth surfacing.
But if you could to not just clicks but something more material (like time on a page, a purchase or social share) into the mix, I think recommendations would get better. Tying into feedback further removed from the initial action, if it could scale, would help the model as well.
The guys from yt explains this in a paper that talks about their recsys.
What they actually do is once you have a candidate set of videos, pick the best ones that maximizes the total time of watch time so they can show you more ads. That's the reason the videos that are frequently recommended are the really long duration ones
Ah. Of course what you optimize depends on what you consider valuable, so that makes total sense.
But if you want to optimize for customer satisfaction, maybe the video watched to completion or the one shared is better than the one the total amount of watch time.
This can be done by leave-one-out cross-validation in a collaborative filtering system. If jane saw schindler's list and gave it 8/10, delete this from the database and use the rest of the data to predict it, then compare the predicted score with the score she gave. I did this for a movie recommendation system i developed: http://www.fmjlang.co.uk/morse/MORSE-Paper.html
Based on the difficulties I had implementing recommender systems, I think it boils down to two main problems:
1. Evaluating recommender systems is REALLY hard. When it comes to evaluation, recommender systems are very different from other machine learning tasks:
Let's assume the classic methodology of collecting a dataset of events, splitting it at a certain point in time to get a training and test set, and checking the precision and recall
- If your recommender system works as expected, you will influence what your users do in the future, so your test set is probably not going to represent what your users would actually do.
- Having a precision and recall of 1 is actually bad. That means your recommender system was perfect, but also useless (you only recommended stuff that the user was already going to pick anyway).
One way to address this is to just use A/B tests and try to optimize some business metric (e.g. number of purchases). This is usually "good enough", but this will make your recommender focus on sales, not on user satisfaction.
There are also some other metrics that can be used[1], but there are so many of them and some are not very practical to implement, so I guess that everyone just goes with the A/B testing approach.
2. Recommendations require an explanation
A lot of recommender algorithms are black boxes.
Sure, you can write "this was recommended based on users similar to you" when you use a collaborative filtering algorithm, but that doesn't help much.
Recommendations without an explanation are not that useful. When a friend recommends you a movie, he'll also tell you "why". Otherwise, it's really hard to make users trust your recommendations (especially if the recommender system recommends something outside of the user's "comfort zone")
I've noticed that a lot of sites have improved on this front though, and I do enjoy those recommendations a lot more.
That's not a sociological explanation. I want something "elephant in the brain"[1]-esque, something that shines light on poor incentive equilibrium on the game theoretic/social plane.
Whether someone likes a movie (or a song) is usually subjective, often emotional, and can at best only be rationalized. There are exceptions, e.g. documentaries. You could e.g. have the system recognize a preference for a director, actor, or genre, and add that to the predicted rating. But i don't think using that kind of information to make a prediction would add to a collaborative system's accuracy.
Look into the history of the Netflix Prize. There were two main factions, those who used only the data provided and those who used metadata about the films.
The Wikipedia page on it covers some issues around the privacy concerns.
Because the current recommender/ML systems model the past as identical to the future as a matter of structural equivalence (There is a profound assumption of linearity between past and future e.g. in time, and linearity in space). They must assume the world is a stable & stationary distribution. Which doesn't leave a lot of room for modeling change.
This actually makes recommender systems things that train humans, that render them more predictable, by grouping them into categories base on 'latent variables'.
It makes everyone more boring by reducing diversity instead of encouraging it.
You can easily see this in your youtube recommendations.
Jaron Larnier calls these kinds of algorithms manipulative - they train people to optimize engagement metrics, rather than sometime qualitative like quality.
A recommender system is super hard because humans are hard. Even if the problem is well defined, like movie recommendations, it is exceptionally hard. Read about the Netflix challenge https://en.wikipedia.org/wiki/Netflix_Prize which took some of the worlds best researchers (trying to win $1MM) 3 years to increase Netflixes algorithm by 10%.
Without any real experience or knowledge beyond dealing with their outcomes in various e-stores, and a bit of thinking on the matter, I'm pretty damn sure the goals are wrong, which leads to the whole setup being wrong. Eg netflix pushes you into having useless ratings that at best become 1 or 5, and nothing in between, by designing themselves in such a fashion: you can't look over your ratings properly, you can't edit them in any sensible fashion, you can't interact or update your ratings at all. And of course, your ratings are limited to whatever the hell they have.
And then because the ratings come out so incredibly divorced from a user's taste, netflix naturally gives up on the rating system, and makes its judgement based on your interactions with the video: how much of a show you watched, how quickly you binged, how quickly you ditched it, etc.
But your viewing habits are not indicative of your taste, and is often quite divorced from it. Everyone has their guilty pleasures; those trash shows you watch because its mindless entertainment. The vast majority of the stuff you watch is not what you wished you watched. That's not the shit you want to be recommended, if you care about being recommended things (otherwise, high popularity is more than sufficient).
But thats because netflix does not have the goal of showing you good movies and shows, things you can watch and actually remember three days later. They have the goal of mindless entertainment (they need your time, not your brain), and the recommender system succeeds at that task. But its difficult to see that goal and think "yeah, thats the personal recommendation I always wanted". It's not what your friend would recommend to you; it's the absolute opposite.
Part of the problem is that a proper catalogue of your ratings for various shows/movies is likely to be a very good indicator of taste, but afaik, the western media has no culture of it, no decent tooling, and no decent systems for it. I know of no decent system to aggregate and curate my movie collection, adding, maintaining, and noting a simplified opinion of it (a rating), for my personal use, beyond the physical/file system storage of it, and half-assed custom-built systems for it (eg text file). Which also implies there's no public dataset of it from which to build a proper recommendation system from.
However, I know at least the anime community offers us a possibility of redemption. MyAnimeList (MAL) and co. do offer a decent system of it, and the community itself lends itself towards the cataloguing task[1]; in my own usage of MAL, I consistently go back through and update my ratings to reflect my current taste, and more importantly, the difference between an old 10-rating and a new one, when I see something that beats out by previous by a long shot, shifting the whole rating. And I'm willing to do the work because maintaining the list is relatively simple, and it offers me great utility; both things you won't find by amazon or netflix.
And I expect others use the site in the same fashion. I think if you really want a decent recommender system, this is the kind of dataset you want. Something that actually happens to reflect a user's taste, because the users themselves are incentivized into maintaining a simplified snapshot of their taste, because they can actually derive value back out of it.
I've had intent for some time now to do that, but never moved beyond the scraping stage (some 40GB of MAL lists sitting on an aws db for almost a year now). Also I'm pretty damn sure with such a dataset, you can get really far with a few simple intuitions (picked up off random papers and my own thinking):
1) User definitions of ratings is mostly arbitrary, but people trivially agree on at least this much: in a 0-10 scale, anything below 5 is disliked, and anything above is liked. 0 is trash, and 10 is good. Normalize on it.
28 comments
[ 2.9 ms ] story [ 76.4 ms ] threadI'm coming at this from the perspective that recommendation engines are, as far as I can tell, practically a hoax.
The reallity is that once you have a subset of items that matches your behavior recsys are learning to rank problems in the broad sense, and any kind of algorithmic feed (fb, Instagram, news, Twitter) is essentially a recsys.
There is a whole conference around these subjects with very good papers and competitions, look at the different kind of approaches to the recsys18 challenge by Spotify at the ACM site
some type of recommendation engines works really well especially the collaborative filtering types.
Exactly what I was asking about, because I'm not so sure. Furthermore, I think it's a category error to include collaborative filtering here, which is not automated/ML/etc.
Netflix < MovieLens < Criticker < Coollector Movie Database
That what we like can't be boiled down to an optimization function over a sparse matrix.
The trick is to find stuff that cuts across the space in some sense—stuff that's out of your current wheelhouse, but still has features that you would like. (E.g. I like the music of both David Rawlings and Mark Ribot, even though they are in wildly different genres, because they both have a certain ear for melody and relationship with dissonance that I like.) I don't think anyone is doing that well yet.
But if you could to not just clicks but something more material (like time on a page, a purchase or social share) into the mix, I think recommendations would get better. Tying into feedback further removed from the initial action, if it could scale, would help the model as well.
What they actually do is once you have a candidate set of videos, pick the best ones that maximizes the total time of watch time so they can show you more ads. That's the reason the videos that are frequently recommended are the really long duration ones
But if you want to optimize for customer satisfaction, maybe the video watched to completion or the one shared is better than the one the total amount of watch time.
1. Evaluating recommender systems is REALLY hard. When it comes to evaluation, recommender systems are very different from other machine learning tasks:
Let's assume the classic methodology of collecting a dataset of events, splitting it at a certain point in time to get a training and test set, and checking the precision and recall
- If your recommender system works as expected, you will influence what your users do in the future, so your test set is probably not going to represent what your users would actually do.
- Having a precision and recall of 1 is actually bad. That means your recommender system was perfect, but also useless (you only recommended stuff that the user was already going to pick anyway).
One way to address this is to just use A/B tests and try to optimize some business metric (e.g. number of purchases). This is usually "good enough", but this will make your recommender focus on sales, not on user satisfaction.
There are also some other metrics that can be used[1], but there are so many of them and some are not very practical to implement, so I guess that everyone just goes with the A/B testing approach.
2. Recommendations require an explanation
A lot of recommender algorithms are black boxes. Sure, you can write "this was recommended based on users similar to you" when you use a collaborative filtering algorithm, but that doesn't help much.
Recommendations without an explanation are not that useful. When a friend recommends you a movie, he'll also tell you "why". Otherwise, it's really hard to make users trust your recommendations (especially if the recommender system recommends something outside of the user's "comfort zone")
I've noticed that a lot of sites have improved on this front though, and I do enjoy those recommendations a lot more.
[1]: Herlocker et al., "Evaluating collaborative filtering recommender systems" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97....
[1] http://elephantinthebrain.com/
The Wikipedia page on it covers some issues around the privacy concerns.
This actually makes recommender systems things that train humans, that render them more predictable, by grouping them into categories base on 'latent variables'.
It makes everyone more boring by reducing diversity instead of encouraging it.
You can easily see this in your youtube recommendations.
Jaron Larnier calls these kinds of algorithms manipulative - they train people to optimize engagement metrics, rather than sometime qualitative like quality.
https://livestream.com/internetsociety2/cybercon/videos/1662...
And then because the ratings come out so incredibly divorced from a user's taste, netflix naturally gives up on the rating system, and makes its judgement based on your interactions with the video: how much of a show you watched, how quickly you binged, how quickly you ditched it, etc.
But your viewing habits are not indicative of your taste, and is often quite divorced from it. Everyone has their guilty pleasures; those trash shows you watch because its mindless entertainment. The vast majority of the stuff you watch is not what you wished you watched. That's not the shit you want to be recommended, if you care about being recommended things (otherwise, high popularity is more than sufficient).
But thats because netflix does not have the goal of showing you good movies and shows, things you can watch and actually remember three days later. They have the goal of mindless entertainment (they need your time, not your brain), and the recommender system succeeds at that task. But its difficult to see that goal and think "yeah, thats the personal recommendation I always wanted". It's not what your friend would recommend to you; it's the absolute opposite.
Part of the problem is that a proper catalogue of your ratings for various shows/movies is likely to be a very good indicator of taste, but afaik, the western media has no culture of it, no decent tooling, and no decent systems for it. I know of no decent system to aggregate and curate my movie collection, adding, maintaining, and noting a simplified opinion of it (a rating), for my personal use, beyond the physical/file system storage of it, and half-assed custom-built systems for it (eg text file). Which also implies there's no public dataset of it from which to build a proper recommendation system from.
However, I know at least the anime community offers us a possibility of redemption. MyAnimeList (MAL) and co. do offer a decent system of it, and the community itself lends itself towards the cataloguing task[1]; in my own usage of MAL, I consistently go back through and update my ratings to reflect my current taste, and more importantly, the difference between an old 10-rating and a new one, when I see something that beats out by previous by a long shot, shifting the whole rating. And I'm willing to do the work because maintaining the list is relatively simple, and it offers me great utility; both things you won't find by amazon or netflix.
And I expect others use the site in the same fashion. I think if you really want a decent recommender system, this is the kind of dataset you want. Something that actually happens to reflect a user's taste, because the users themselves are incentivized into maintaining a simplified snapshot of their taste, because they can actually derive value back out of it.
I've had intent for some time now to do that, but never moved beyond the scraping stage (some 40GB of MAL lists sitting on an aws db for almost a year now). Also I'm pretty damn sure with such a dataset, you can get really far with a few simple intuitions (picked up off random papers and my own thinking):
1) User definitions of ratings is mostly arbitrary, but people trivially agree on at least this much: in a 0-10 scale, anything below 5 is disliked, and anything above is liked. 0 is trash, and 10 is good. Normalize on it.
2) Anim...
[0] https://github.com/javascript-machine-learning/movielens-rec...