Show HN: I trained an AI model on 120M+ songs from iTunes (maroofy.com)
I just shipped a project I’ve been working on called Maroofy: https://maroofy.com
You can search for any song, and it’ll use the song’s audio to find other similar-sounding music.
Demo: https://twitter.com/subby_tech/status/1621293770779287554
How does it work?
I’ve indexed ~120M+ songs from the iTunes catalog with a custom AI audio model that I built for understanding music.
My model analyzes raw music audio as input and produces embedding vectors as output.
I then store the embedding vectors for all songs into a vector database, and use semantic search to find similar music!
Here are some examples you can try:
Fetish (Selena Gomez feat. Gucci Mane) — https://maroofy.com/songs/1563859943 The Medallion Calls (Pirates of the Caribbean) — https://maroofy.com/songs/1440649752
Hope you like it, and would love to hear any questions/feedback/comments! :D
438 comments
[ 3.8 ms ] story [ 339 ms ] threadAlso if it's not a trade secret can you expand "My model analyzes raw music audio as input and produces embedding vectors as output." What kind of analysis are you doing? What is the criteria for picking recommendations? I always find this area of music analysis intersting.
But not sure if the song I input and the songs I get back are similar. Going to try them out further.
What things are they supposed to be similar at? beats? melody?
But for now, ended up using plain old FAISS.
Also this song, Shangri-La Is Calling, is bugging out: https://maroofy.com/songs/1632142336
Most other services try to find matches by seeing what other songs the people who like the searched song like, and finding trends amongst those. This site finds songs with similar sounds and rhythms by looking in the vector space. Awesome! Congrats on the finished site!
Curious what sort of metrics are taken into account, and if any supervision is provided.
Apple might as well buy your project for iTunes since they are extremely behind in anything AI.
Take Raga's Dance by Vanessa-Mae, A R Rahman, ... Royal Philharmonic https://maroofy.com/songs/476841571 . I put in this track expecting other fusion songs to pop up, and arguably some do, but much more often it feels like a 20 second section was used to define the original song and it misses the underlying concept. Like it got, in my subjective description, the epic violin in orchestral music, but it completely ignores the fusion between the distict styles of traditional indian singing/instrumentals and western ochestral and also ignores the call response structure between the violin and carnatic players, which is the what I actually care about. Other songs have the vocals but no epic backing. It feels like it's matching multiple samples from the song instead of the whole song.
This feels very promising since it clearly is picking up the styling of the specific songs across different genres and languages. I look forward to seeing where this goes.
I also think it would be interesting if there was a way to specify two different songs to find either only the common things and/or to find what the fusion of those two tracks produces.
But the model architecture I'm using is kinda outdated as well, gotta iterate on it more to improve it further!
I'm also thinking of letting users upvote/downvote results, which can also help improve quality on the ranking side.
Yes, it's hard to find a song I really really like, but 1-in-10 seem to be something I'd add to my eclectic "thumbs up" playlist. And almost none of them are by any artist that I've heard of before.
This is huge for me. Thanks.
If you need someone to test your model, you will never find one with more eclectic/strange taste than me ;)
1) https://maroofy.com/songs/1443834381
2) https://maroofy.com/songs/714207207
Nine Inch Nails
https://maroofy.com/songs/1440934933
Recommended Muppets
Rather than get back anything "acoustically similar" it simply returned a list of other songs on the same album (several of which are far from being acoustically similar).
No drama, you're attempting to cover a lot of ground, but I'm guessing there was no actual fingerprint there for that work and no sense of other songs that sounded similar.
ADDENDUM: Okay, I had to select the song <doh> .. but still "something went wrong" - perhaps hugged to death or not found to process. No matter :-)
I agree with the other commenter - this is huge for me. Please, do whatever you need to do to monetize this so it never goes away. I would love to pay you for this.
If you listen to a music with a real intro, it gives strange results. For example: "Goodbye Blue Sky - Pink Floyd" (https://maroofy.com/songs/1065976153)
Spotify on the other hand seems to want to send me to the same group of artists and tracks I've listened to before, following some Collatz conjecture type algorithm that eventually converges on the same tuned playlist for that genre, no matter what the starting parameters may be.
What makes it different from a big "play me a random song" button then?
> it seems to identify similar tempo and melodic riff
Era? Artist? Genre? Sound? Tempo?
Personally I spend my time finding similar-era music because I like to hear how sounds evolved.
See this paper from https://everynoise.com/ : https://everynoise.com/EverynoiseIntro.pdf
IIRC they try to classify music on 17 different points/features. What you see on the web is an attenpt to visualise (and provide a guide to music based on) some of them
Echo nest was great for its time, but if they have kept up, they're not exposing their more modern learned features to users anymore.
I'm not at liberty to say what, sadly, as I work for Spotify.
I think I can say that one of the main challenges is running this analysis for users. It's prohibitively expensive (or was prohibitively expensive) to use this to keep track of and run recommendations for what users are listening for each user.
It can be used on smaller scales, but, well, it's probably NDA :)
I def prefer for that common index to have a permissive license though!
But I've been watching them, I will speculate. A few years ago, Spotify had two young interns, Sander Dieleman and Aäron van den Oord. We know a bit of what they worked on, because Dieleman blogged on it, and indeed it was something a lot like what OP has made here - only better, I would say. I asked him, and Dieleman was allowed to say that the thing they built was one of the inputs into the then-new Discover Weekly, which made headlines for how outrageously good it was.
But Dieleman and v.d.Oord did not stay at Spotify. They were headhunted by DeepMind, and have had a VERY impressive track record there over the years.
And I wonder why. Was there a conflict between the old school ML of the Echo Nest people and the new fancy neural net kids? Or was it just, as GP alludes to, that the NN methods were just too computationally expensive and they failed to justify their costs to leadership?
I honestly don't know much about recommendations (and what I know I probably cannot tell). But there's definitely continuous work done on them. But it can also be hampered by extremely conflicting requirements (where "some" both means double-digit procent of users and these "some"s overlap with each other):
- some users want more of the same, some users want a more diverse listening experience. Some of these users are the same user, but on different days
- some users mostly prefer curated suggestions, some users want ranodm stuff. They can also be the same user :)
- some users a heavily weigted to only a few artists, some users listen to evereything and anything. And even this can be the same user :)
- there's probably stuff about licensing, availability, contracts etc. at play as well, because in streaming services it's always there, in very bizarre ways
Basically every single tweak to recommendations will break them. And yeah, Spotify employees will complain about this more than anyone else, all the time :)
Examples:
The Oblio Joes - "Captain of the Moon"
The Bondage Fairies - "Levenus Supremus"
... both chosen so "Just shove a bunch of recent pop-rock at the user" won't work.
Feels a little bit like fortune telling, I guess it is, in the sense that I am listening closely to what makes the songs similar, not just listening, but actively trying to find the similarities, so even a couple notes in progression, or drum-beats and I'll say oh, yes, that matches.
Finds very different music, not necessarily what I'd listen to in many cases, but kudos for getting me clicking through a decent pile before going wait, that's a nope, you're grasping AI-type-being.
The first track I searched for was "Hazmat Modine - Bahamut" (https://maroofy.com/songs/253108933), it seems to recommend things with some similar instruments (e.g. brass and sax) but not really similar taste or style.