“Our recommender system aims at maximizing user satisfaction and, as a proxy, we consider the completion count (and rate) at the session level, i.e., the amount of tracks recommended by the policy that are completed by the user. For this task we first train an agent offline using a non-sequential world model (CWM) as user simulator, and then deploy the agent online to serve recommendations to the users.
The outcomes of our user model are primarily consumption-focused and summarize the probability of user-item interactions. Specifically, CWM has been optimized for three targets: completion, skip and listening duration. The reward for the agent is computed as the sum of the probability of completion for each track in an episode.
…
All policies take as inputs features of the user who made the request and the pool-provided set of tracks with their associated features. The goal of each policy is to select and order a list of tracks from the pool that maximizes expected user satisfaction, which we measure by counting the number of tracks completed by the user.”
Came here to excerpt that same snip and to comment that maybe I’m an outlier but based on my own behavior I’d want to reinforce the model’s propensity to suggest tracks that the user likes/favorites/saves/adds to library (all past names for the same action due to some INSANE UX struggle going on at Spotify, topic for another rant) or adds to a playlist, or plays again outside of the context of a recommendation playlist. I think, for my purposes, ‘plays to completion’ is a weak proxy for whether I liked the track, as I’ll sometimes just put on a recommended playlist and leave the room, or tune out and work, etc.
I’ve thought this exact thing many times. However, I wonder if they do this because they’re not actually optimizing for user satisfaction, but for complete plays, for financial reasons.
Financially, I think they count a stream after 30 seconds, and they pay per stream, therefore they'd either want you to skip before 30 seconds (making it free for them) or otherwise spend the maximum time with the song (such that they only pay for 1 stream).
The most expensive way for you to listen to spotify is to skip the song every 31 seconds.
This is not how they pay out subscription $. They do not pay per stream. While the payout algorithm is highly complicated, essentially it is a % of subscription dollars distributed pro-rata by stream count to copyright holders (where a 1 stream is at least 30s of listening).
Consumption is directly linked to ultimate payments by Spotify to copyright holders. However it is not done on a per stream basis. It is paid out of a pool of earnings on a pro-rata basis. Each copyright holder negotiates its own formulas with Spotify / others for payouts, however they all have some version of the pro-rata payout system. So in the sense of direct financial incentive, it would be best to drive customers toward music where the payout formulas are the lowest (e.g. their owned content), assuming it wouldn’t increase churn or reduce premium sign ups.
No, complete plays is simply their measure of user satisfaction. If I like a song, I usually listen to it until the end. If I do not like a song, I usually skip it after a few seconds.
Said upthread, a much better metric would be use of the song outside of the generated playlist. Added to other playlists, added to library, listened to again hours or days later of your own accord, etc. Basically, in some way tied to a user action rather than the absence of one.
I’m fairly open to music, I’ll likely listen to a song once if it’s on a playlist. But that doesn’t mean I liked the song, only that I gave it a shot. It certainly means I liked it more than the song I just skipped, sure, but it also doesn’t mean I necessarily want to hear it again.
On these AI generated playlists I'll actually skip songs even if I like them because I'm just cruising through the list, then might go and save a full album from the band to check out later, in full. So the selected behavior is really not representative in my case.
Not when it's recommending something it _should_ already know I like (and probably why it added it to the list). Also, sometimes, I may like something, but not add it to my personal list. Since there's no way to rate a song, I use likes only for songs I really like.
I think the real point to be made here is that this is part of the inner workings of the system, that most users of the system are unaware of. Hell, this article and the ensuing discussions do not leave it completely clear how much of the system works. Like, I despise that when I create a new playlist with a name, it recommends a bunch of tracks based on the name of the playlist. Sometimes that'll be tracks with words in the name of the playlist in their name or some other odd metric, like it'll add songs from an artist that has a song that happens to be the name of the playlist. If you don't puzzle this out for yourself, you're possibly creating a very UN-optimized playlist for yourself.
I think complete plays is a pretty fair metric, though it hurts 45 minute long epics or Dj sets or w/e.
The bigger issue I have is with “number of times played.” I listen to some comfort-food music over and over again, but the most rewarding music I listen to is challenging in some way and maybe isn’t something I listen to over and over.
Complete plays is a fair metric, if it doesn't take into consideration related traits like when I love a particular remix of a song that is very different than the original, the system decides that I loved the original song and is now going to recommend songs similar to or liked by other users of the system that liked that original song.
This is another place where Pandora really set themselves apart, the Music Genome Project. Any given track that went through curation has a (possibly very) large set of attributes assigned to it. This song you liked, it has a heavy bass-line, noticeable amount of shuffle, light drums, syncopated rhythms, etc. That's far better (to me) than "you might like other songs by this artist" or "other listeners of this artist also listen to", where the last one gets really sketchy when there's not a lot of listeners for the artist.
I'm also curious how it treats listening to a track on a Spotify station that is mixed, where they transition in to and out of the track late or early, so you won't hear the full track, does that still count?
Heck, sometimes I'll get almost to the end of the track and skip to the next just because the track has a long tail and I want to get to something that has more energy, not the dwindling remains of the rhythm or some soft piano fade out at the end of a 130 BPM track that had a lot of energy throughout most of it.
If they're going to make all of these sometimes seemingly arbitrary judgements of whether or not I like something based on these weird things like "there's a common English word in the playlist title that also shows up in all these song titles we're going to recommend to you", at least a list or chart of how it works somewhere would be nice, so I can make more effective use of the system.
I have a feeling Spotify is trying to make the UI as different to Apple as possible so that the switching cost on the UI will be huge and people will not be comfortable to switch. This is why Spotify will choose some weird UI.
I'm never going to be satisfied with Spotify's recommendations until it's easy for me to curate the input to the recommender.
As things stand, to prevent Spotify from giving me crap recommendations, I have to keep secrets from it! If I want to listen to Led Zeppelin once but don't want Spotify to think "oh I should recommend classic rock from now until the end of time", then I have to go to a different app (YouTube or whatever).
There's a "private session" feature in Spotify which allows me to prevent searches and listens from entering into history, but Spotify reallyreally doesn't want me to use it — it's cumbersome to access and it gets turned off automatically after a few hours.
EDIT:
The satisfaction achieved by whole-account recommendations determined by algorithm described in this post is going to be crushed by some app which gives me a UI where I can say "give me recommendations based on these N arbitrary things". Not just "this one artist" or "everything in my history". Let me design my own recommended channel.
Add to that having kids at home that sometimes listen to Paw Patrol and you will end up with a daily mix that consists of (in my case): Death Metal, news, random clip from paw patrol, German rap (I only like one artist not that genre in general).
I’ve been longing for a “kids mode” for a long time.
I gave my kid it's own Spotify account (we have a family subscription), and I installed "island" in android enabling me to install a separate Spotify instance logged into their account. Now I have 2 Spotifys installed in parallel, one for me and one for them.
I find that spotify will tend to just smash music from my other playlists and other history into that station. I can never get it to recommend me new stuff.
The autogenerated genre playlists have this issue too. "House Music" is a very broad genre (or metagenre, given how many subgenres exist), so you'd think that they wouldn't end up being largely comprised of tracks I already have on my own playlist. And the rest end up being kind of bland and not really quite what I'm looking for.
I seem to remember that Spotify used to have a heavy focus on employee-operated playlists that curated new releases, though I didn't really take advantage of them at the time. The newer (cheaper) algorithm-driven ones don't really cut it.
They removed this feature. Now there is a psuedo-replacement called "enhanced shuffle" which interleaves recommended songs in your shuffled playlist. Unfortunately, this drastically reduces the amount of song discovery per unit time.
What I want to listed to depends greatly on my mood. Might be classic rock, might be jazz, might be contemporary pop, might be alternative or punk or metal or even classical.
Might also depend on who else is in the room or car. If I'm playing music for myself my selections will be different from what I'd play if my wife was listening, because our musical tastes are different. Or a group of friends, etc.
I've never found that recommendations or automatic playlists are very good, on any of the services, though my experience is mainly with YouTube music.
I used to really like Pandora's a lot. So much so that I once emailed them commending them on this and they sent me a t-shirt. If memory serves the difference was that Pandora's associations were human curated.
Just let me exclude songs from recommendations, please. There are songs that are superficially similar to those I enjoy, but that I never want to hear. It's infuriating.
Maybe I’m misunderstanding something fundamental, but I’m struggling to see the reason for this work. They have a (presumably) supervised model that estimates the likelihood that a user will like a song. Then they train an RL model using the supervised model as an oracle. They note that the optimal result for the RL model is just making the same prediction as the supervised model.
What are we doing here folks? Training models to match other models? Why? That’s good enough to get into KDD?
- the offline model data has a non-sequential world model
- the offline model is a great starting point but slow to react to realtime changes in user preferences
- the in-streaming of new data points makes the reward prediction inherently sequential
- hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point
The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data)
If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky.
If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency.
Spotify's rating and recommendation system leaves quite a bit to be desired. They need granularity, and they need to explain the algorithm to some extent.
If I like a track what exactly am I liking? The genre? The artist? Only specifically that song? Some maths that approximate a certain properties in the track? Some combination? How does liking song X influence Spotify's recommendation system on average? I have experienced the same as other posters WRT listening to a genre for a session only to have Spotify fixate on that genre until I've skewed the average in another direction. Small hack I've found if you make a playlist with a large and diverse sample size you can nudge their playlist enhancer to stay with the mood you are going for.
On granularity, I would want to be able to say:
Let x = track, genre, artist
I don't like X, never play it.
Exclude X just for this listening session.
Skew toward more X
I'd also imagine this would produce much richer data for future recommendation systems. If Spotify were to train on the data I've generated on their platform it would end introducing bias by the nature of me working around their systems and still not getting what I want, granted I may be an outlier.
I‘m shocked how unbelievable bad the recommendations from Spotify are.
They never were any good, but they also don’t improve, despite how often they brag about some new recommendation engine, feature or improvement on their engineering blog.
A multi billion dollar company that fails so hard at this.
The best recommendations I got were from Pandora, unbelievable good and on point and that was 15 years ago. I can not use them anymore in Europe so I m stuck with Spotify.
Maybe this GPT4 based project will give me better recommendations.
I tried out the new "Personal DJ" mix on Spotify last week and bounced when the way over the top AI voice stated my username as a part of the most cringe inducing intro. The feature was awful and there wasn't even any notion of fast-forwarding through the bad parts. 0/10. Will not try again.
I agree so much with this. I was excited for the feature until the first 10 seconds. I don't want a fake DJ "talking to me" or attempting to emulate a radio DJ. It's not like all I have to look at is a dial with numbers on it. I can see all the details of what you're playing, thanks. Shut up and just let me see if I like what you think I'll like, enough that it makes me feel that your recommendations won't be immediate rejections. Or just have the dumb AI talk to me about it, so I know that I absolutely do not want to use the feature. sigh
That makes me a doomsayer, basically, but I consider this to be evidence that every single aspect of our lives will eventually be somehow controlled by AI.
Sure, you can claim that there's always the choice of not doing what the AI suggests, but that'll not apply to the kids growing up in a world where they're being breast-fed literally everything, including their playlist.
And their reason will be "because it's good", which is related to "because it's fun". When AI is perfectly pleasing, people will be fully enslaved by fun.
Fun already seems to be the driving factor for most people, so I'm not sure how to find anything positive in any of this.
Disclaimer: At the peak, I had around 300 gigs worth of language models stored on my machine. Now I'm down to around 120. :p
41 comments
[ 4.2 ms ] story [ 90.6 ms ] threadThe outcomes of our user model are primarily consumption-focused and summarize the probability of user-item interactions. Specifically, CWM has been optimized for three targets: completion, skip and listening duration. The reward for the agent is computed as the sum of the probability of completion for each track in an episode.
…
All policies take as inputs features of the user who made the request and the pool-provided set of tracks with their associated features. The goal of each policy is to select and order a list of tracks from the pool that maximizes expected user satisfaction, which we measure by counting the number of tracks completed by the user.”
Although, I suppose this line of thinking makes sense for ad-supported streaming, which is likely used by the majority of users.
The most expensive way for you to listen to spotify is to skip the song every 31 seconds.
How would you measure user satisfaction?
I’m fairly open to music, I’ll likely listen to a song once if it’s on a playlist. But that doesn’t mean I liked the song, only that I gave it a shot. It certainly means I liked it more than the song I just skipped, sure, but it also doesn’t mean I necessarily want to hear it again.
So you don’t add them to your library?
I think the real point to be made here is that this is part of the inner workings of the system, that most users of the system are unaware of. Hell, this article and the ensuing discussions do not leave it completely clear how much of the system works. Like, I despise that when I create a new playlist with a name, it recommends a bunch of tracks based on the name of the playlist. Sometimes that'll be tracks with words in the name of the playlist in their name or some other odd metric, like it'll add songs from an artist that has a song that happens to be the name of the playlist. If you don't puzzle this out for yourself, you're possibly creating a very UN-optimized playlist for yourself.
The bigger issue I have is with “number of times played.” I listen to some comfort-food music over and over again, but the most rewarding music I listen to is challenging in some way and maybe isn’t something I listen to over and over.
Basically, Spotify’s metrics aren’t my metrics.
This is another place where Pandora really set themselves apart, the Music Genome Project. Any given track that went through curation has a (possibly very) large set of attributes assigned to it. This song you liked, it has a heavy bass-line, noticeable amount of shuffle, light drums, syncopated rhythms, etc. That's far better (to me) than "you might like other songs by this artist" or "other listeners of this artist also listen to", where the last one gets really sketchy when there's not a lot of listeners for the artist.
I'm also curious how it treats listening to a track on a Spotify station that is mixed, where they transition in to and out of the track late or early, so you won't hear the full track, does that still count?
Heck, sometimes I'll get almost to the end of the track and skip to the next just because the track has a long tail and I want to get to something that has more energy, not the dwindling remains of the rhythm or some soft piano fade out at the end of a 130 BPM track that had a lot of energy throughout most of it.
If they're going to make all of these sometimes seemingly arbitrary judgements of whether or not I like something based on these weird things like "there's a common English word in the playlist title that also shows up in all these song titles we're going to recommend to you", at least a list or chart of how it works somewhere would be nice, so I can make more effective use of the system.
I've felt this exact same sentiment and can't help but wonder if it's them being partially afraid of Apple Music in some way.
Since Apple Music launched with "add to library" as their shortcut action and "love" buried a bit.
I'm never going to be satisfied with Spotify's recommendations until it's easy for me to curate the input to the recommender.
As things stand, to prevent Spotify from giving me crap recommendations, I have to keep secrets from it! If I want to listen to Led Zeppelin once but don't want Spotify to think "oh I should recommend classic rock from now until the end of time", then I have to go to a different app (YouTube or whatever).
There's a "private session" feature in Spotify which allows me to prevent searches and listens from entering into history, but Spotify really really doesn't want me to use it — it's cumbersome to access and it gets turned off automatically after a few hours.
EDIT:
The satisfaction achieved by whole-account recommendations determined by algorithm described in this post is going to be crushed by some app which gives me a UI where I can say "give me recommendations based on these N arbitrary things". Not just "this one artist" or "everything in my history". Let me design my own recommended channel.
I’ve been longing for a “kids mode” for a long time.
I seem to remember that Spotify used to have a heavy focus on employee-operated playlists that curated new releases, though I didn't really take advantage of them at the time. The newer (cheaper) algorithm-driven ones don't really cut it.
Might also depend on who else is in the room or car. If I'm playing music for myself my selections will be different from what I'd play if my wife was listening, because our musical tastes are different. Or a group of friends, etc.
I've never found that recommendations or automatic playlists are very good, on any of the services, though my experience is mainly with YouTube music.
What are we doing here folks? Training models to match other models? Why? That’s good enough to get into KDD?
- the offline model data has a non-sequential world model
- the offline model is a great starting point but slow to react to realtime changes in user preferences
- the in-streaming of new data points makes the reward prediction inherently sequential
- hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point
The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data)
If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky.
If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency.
It's a good achievement.
If I like a track what exactly am I liking? The genre? The artist? Only specifically that song? Some maths that approximate a certain properties in the track? Some combination? How does liking song X influence Spotify's recommendation system on average? I have experienced the same as other posters WRT listening to a genre for a session only to have Spotify fixate on that genre until I've skewed the average in another direction. Small hack I've found if you make a playlist with a large and diverse sample size you can nudge their playlist enhancer to stay with the mood you are going for.
On granularity, I would want to be able to say:
Let x = track, genre, artist
I don't like X, never play it.
Exclude X just for this listening session.
Skew toward more X
I'd also imagine this would produce much richer data for future recommendation systems. If Spotify were to train on the data I've generated on their platform it would end introducing bias by the nature of me working around their systems and still not getting what I want, granted I may be an outlier.
Edit: Formatting & typo
A multi billion dollar company that fails so hard at this.
The best recommendations I got were from Pandora, unbelievable good and on point and that was 15 years ago. I can not use them anymore in Europe so I m stuck with Spotify.
Maybe this GPT4 based project will give me better recommendations.
https://github.com/Railly/spotigen-chat-gpt-plugin
0/10 indeed
Sure, you can claim that there's always the choice of not doing what the AI suggests, but that'll not apply to the kids growing up in a world where they're being breast-fed literally everything, including their playlist.
And their reason will be "because it's good", which is related to "because it's fun". When AI is perfectly pleasing, people will be fully enslaved by fun.
Fun already seems to be the driving factor for most people, so I'm not sure how to find anything positive in any of this.
Disclaimer: At the peak, I had around 300 gigs worth of language models stored on my machine. Now I'm down to around 120. :p