Spotify can shut down any day. Even if it survives, it's removing content all the time. How are future generations supposed to study and listen to music if it is lost? Imho, someone has to do it.
The idea is that the streamers and major labels cannot be trusted to keep this available for future generations, so if we want to preserve our shared culture we should take matters into our own hands.
I think the negatives for artists are minimal while the benefits of preserving a annotated snapshot of contemporary music for future generations is very valuable.
It is not hard. But please don't misuse it and ruin the fun for everyone. It is nice to be able to use the music relatively easily for hobby projects. My music server has functionality to play tracks from Spotify this way:
"at scale" could mean they had direct access to a server or to storage, maybe because they had an insider giving them access, or they found secrets that had leaked somewhere?
I definitely was not aware Spotify DRM had been cracked to enable downloading at scale like this.
The thing is, this doesn't even seem particularly useful for average consumers/listeners, since Spotify itself is so convenient, and trying to locate individual tracks in massive torrent files of presumably 10,000's of tracks each sounds horrible.
But this does seem like it will be a godsend for researchers working on things like music classification and generation. The only thing is, you can't really publicly admit exactly what dataset you trained/tested on...?
Definitely wondering if this was in response to desire from AI researchers/companies who wanted this stuff. Or if the major record labels already license their entire catalogs for training purposes cheaply enough, so this really is just solely intended as a preservation effort?
Id be stunned if we didn't find out Anna's Archive is a front for a handful of shadier VCs who are into AI. Even if AA themselves don't know it and just take the cash.
> Spotify itself is so convenient, and trying to locate individual tracks in massive torrent files of presumably 10,000's of tracks each sounds horrible.
Download the lot to a big Nas and get Claude to write a little fronted with song search and auto playlist recommendations?
> The thing is, this doesn't even seem particularly useful for average consumers/listeners
Yeah. To me it is not really relevant. I actually was not using spotify
and if I need to have songs I use ytldp for youtube but even that is
becoming increasingly rare. Today's music just doesn't interest me as
much and I have the songs I listen to regularly. I do, however had, also
listen to music on youtube in the background; in fact, that is now my
primary use case for youtube, even surpassing watching movies or anything
else. (I do use youtube for getting some news too though; it is so sad that
Google controls this.)
The first users of this dataset will be Big Tech corps. Meta, Alphabet, OpenAI, Microsoft, Apple will all be happy to use this dataset for training their LLMs.
Flippant response: If it's ok for Meta for commercial use, why not for researchers for legitimate research work?
More serious response: research is explicitly included in fair use protections in US copyright law. News organizations regularly use leaked / stolen copyrighted material in investigative journalism.
> The thing is, this doesn't even seem particularly useful for average consumers/listeners, since Spotify itself is so convenient, and trying to locate individual tracks in massive torrent files of presumably 10,000's of tracks each sounds horrible.
Are you aware Annas Archive already solved the exact same problem with books?
Basically there is an entity between Annas Archive and the torrents: hosters. AA has searchable metadata and a hash value. The hosters keep track of hash values, the cached files and in which torrents they are backed up, and take up almost the entire legal liability. Users search on AA what they are looking for but ultimately download it from a hoster.
This, indeed, has mostly implications for ML, training, etc. As otherwise the whole catalog is available to partners, but costs a lot. So Anna did indeed liberate the content, but I'm definitely not switching off my Spotify subscription, even though, in my personal taste, neither quality, nor UI does match Apple Music. It is still useful to have s.o. serve the content for you.
I believe that we need to distinguish between convenience and preservation here. It is indeed convenient for consumers to use Spotify now whilst it exists and operates the way it does. They could go under, they could change their business model, they could decide to purge everything that is not easily justifiable commercially.
As a society, we should do our best to preserve this trove.
A little off topic, but I remain naively hopeful that the horror you describe will keep Spotify from going down the same road Netflix did once content owners decided to get into the streaming business themselves, so that streaming a movie today requires you to "change the channel" to whichever service offers that movie.
Can you imagine your favorite playlist needing to swap among 10 apps, each requiring a $10/month subscription?
This makes me think that after the crack, they probably had to come up with a formula that can statistically calculate how fast they should download spotify songs without letting Spotify realizing that they're scraping the company data and block the access. Remind me of Alan Turing formula after cracking the Enigma
To put this into perspective, What.CD [0] was widely considered to be the music library of Alexandria, unparalleled in both its high quality standard and it's depth. What had in the ballpark of a few million torrents when it got raided and shut down. Anna's rip of Spotify includes roughly 186 million unique records. Granted, the tail end is a mixed bag of bot music and whatnot, but the scale is staggering.
about the scale, the same album in the tracker had several submissions, for dedicated format and regional editions.
while one can compare in terms of number of tracks, the quality used to be in another level altogether. from the article:
> The quality is the original OGG Vorbis at 160kbit/s.
meanwhile the tracker had 16/24-bit flac rips of vinyl, with decent quality control where the track's metadata was verified for any artifacts. for the given quality, one could rip youtube music (maybe not as easily anymore) and achieve a larger scale in a similar quality level.
now if hypothetically tidal had all the music of the world and was accessible this way, then it would be a comparable resource. insane regardless.
It's 160kbit/s for popularity>0 and 75kbit/s for popularity=0. I'm surprised Anna's Archive went for this given that these are not archival quality bitrates. It appears they did this because they found a way, rather than seeking to create a library of music.
my comparison was with source catalog in spotify compared to the private tracker. spotify is working on the high fidelity mode, but so far it is not rolled out worldwide.
i completely understand the archive's decision on applying their own compression.
> What.CD [0] was widely considered to be the music library of Alexandria, unparalleled in both its high quality standard and it's depth.
It was quality in technical quality of the audio in the files, but also in the organization and sourcing of the material, the QA-process of the encoding - down the the specific release the audio-file was from.
There was quantity, sure, but that was secondary to the quality. The quantity was just a side-effect of the place being known for quality, making it an attractive arena to participate in.
And it also had all the "weird"/non-standard things you don't find on mainstream streaming-services precisely because that is what independent curators are good at and often driven by.
This Anna's release... While in itself impressive in many ways does not compare to the things What.CD represented. It's almost the exact opposite:
- focus on most popular content - niche content (even by mainstream Spotify-standards) is not included
- quality is 160kbps ogg files, which is far from lossless, it's not tightly coupled to a release and even as so far the audio-grading goes, there's no transparent QA process for the content, nor is it available in audiophile fidelity.
Moral and legal discussion aside, this is technically very impressive. I also wouldn’t be surprised if this somehow kickstarts open source music generative AI from China.
For electronic music, it's around the lowest bass root note that most systems can play well without a subwoofer. C pretty much requires a sub and things rarely go lower than that.
I have Spotify premium but the constant shuffle of content availability has meant I’ve stared routinely archiving my liked songs to avoid any rug pull. Zspotify and co still work a charm.
Not that we should, but it's technically feasible to have a music streaming server with the torrent as the backend, and selectively download the part of the torrent in respond to on-demand streaming request from the client.
Unrelated, but I just can't stop myself from saying that I absolutely hate Spotify even though I'm a paying customer. Fuck you Spotify. You were supposed to be a convenient way to discover and listen to music. Now you are only convenient for listening to music, and absolutely terrible for any recommendations. This is sad really. Spotify had good recommendations. It's absolutely in a position where it can provide good recommendations — it has both a vast music library and a vast amount of data on user preferences. And it chooses to push procedural/ai-generated slop instead to earn more money. I thought that maybe buying $SPOT stock will make me more at peace with its greed, but it didn't work. Spotify fucking deserves to crash and burn because it sees paying customers as idiots who might not notice they are fed garbage. Fuck you Spotify, fuck you.
Why do you want a megacorp to tell you what to listen to!?? There are a million ways to do discovery where some enshitified corp isn’t incentivized to push something at you.
When they launched Discover Weekly thing, I used to add at least 1 track from it to my library - it was insanely good. Now it's all junk - not even close to what I listen to.
They also removed a lot of discovery features - Playlist Radio - for example. And they still do have some version of it on the backend, but you have to go through some weird mechanisms to trigger it - like play the last song in playlist, wait till it ends (or rewind) and you get the playlist radio. But it's also a crippled version of it - prefers playing the exact same popular songs for some reason.
Then they released this DJ thing, which is laughably bad. No Spotify, I don't want someone talking to me with useless information in between songs. Who though that was a good idea? Who actually uses that?
There hasn't been a change in Spotify in last 7 years or so that wasn't negative.
Holy crap. This is going to trigger a five-alarm fire at Spotify Engineering. This has got to be among the largest proprietary datasets ever unintentionally publicized by a company.
I recall many interesting tracks that were very aggressively deleted from all platforms in sync. I wonder if I could find them in this archive.
There is contemporary lost media being created every day because of how we distribute things now. I think in some cases, the intent of the publisher was to literally destroy every copy of the information. I understand the legal arguments for this, but from a spiritual perspective, this is one of the most offensive things I can imagine. Intentionally destroying all copies of a creative work is simply evil. I don't care how you frame it.
Making media effectively lost is not much different in my mind. Is it available if it's sitting on a tape in an iron mountain bunker that no one will ever look at again?
Uh, cool, I guess? I want to applaud that, but, first off, unless you are OpenAI or Facebook, it is not exactly plausibly easy to participate in the festivities. Even if I had spare 300 TB laying around, how the fuck do I download that?
But, more importantly, I cannot even say "good for you", because I don't actually think it is good for Anna's Archive. I wouldn't touch that thing, if I was them. Do we even have any solid alternatives for books, if Anna's Archive gets shot down, by the way? Don't recommend Amazon, please.
I really don't understand how focusing on source quality files is supposed to be a "major issue" with the music preservation community. It's bizarre for them to talk about these being barriers for creating a "full archive of all music that humanity has ever produced" have and their answer be scraping Spotify to end up with a music library comprised of many AI and bulk produced songs at 75/160kbps.
great. Spotify just removes things all the time (things I actively listen to and work on for my jazz practices, one day just go "poof" because they didn't want to pay the record company anymore), and they are not as a company deserving of the role of "keeper of all the world's music". They don't give a shit and they'd vastly prefer we all listen to their AI generated royalty free crap and Joe Rogan.
It seems to be that the metadata doesn't include the lyrics, probably because they are provided by Musixmatch. It would have been nice to have a database of lyrics linked to ISRCs. AFAIK Lrclib doesn't support downloading lyrics for a given ISRC.
193 comments
[ 4.0 ms ] story [ 123 ms ] threadwhat would your ISP do?
I think the negatives for artists are minimal while the benefits of preserving a annotated snapshot of contemporary music for future generations is very valuable.
> A while ago, we discovered a way to scrape Spotify at scale.
They wont and shouldn’t divulge the details, but I imagine that would be a fun read!
https://codeberg.org/raphson/music-server/src/branch/main/sp...
I definitely was not aware Spotify DRM had been cracked to enable downloading at scale like this.
The thing is, this doesn't even seem particularly useful for average consumers/listeners, since Spotify itself is so convenient, and trying to locate individual tracks in massive torrent files of presumably 10,000's of tracks each sounds horrible.
But this does seem like it will be a godsend for researchers working on things like music classification and generation. The only thing is, you can't really publicly admit exactly what dataset you trained/tested on...?
Definitely wondering if this was in response to desire from AI researchers/companies who wanted this stuff. Or if the major record labels already license their entire catalogs for training purposes cheaply enough, so this really is just solely intended as a preservation effort?
Download the lot to a big Nas and get Claude to write a little fronted with song search and auto playlist recommendations?
Yeah. To me it is not really relevant. I actually was not using spotify and if I need to have songs I use ytldp for youtube but even that is becoming increasingly rare. Today's music just doesn't interest me as much and I have the songs I listen to regularly. I do, however had, also listen to music on youtube in the background; in fact, that is now my primary use case for youtube, even surpassing watching movies or anything else. (I do use youtube for getting some news too though; it is so sad that Google controls this.)
For them, 300TB is just cheap
More serious response: research is explicitly included in fair use protections in US copyright law. News organizations regularly use leaked / stolen copyrighted material in investigative journalism.
Are you aware Annas Archive already solved the exact same problem with books?
As a society, we should do our best to preserve this trove.
Can you imagine your favorite playlist needing to swap among 10 apps, each requiring a $10/month subscription?
Curious why not? Assuming you only used the metadata. I think they would be considered raw facts and not copyrightable.
[0] https://en.wikipedia.org/wiki/What.CD
while one can compare in terms of number of tracks, the quality used to be in another level altogether. from the article:
> The quality is the original OGG Vorbis at 160kbit/s.
meanwhile the tracker had 16/24-bit flac rips of vinyl, with decent quality control where the track's metadata was verified for any artifacts. for the given quality, one could rip youtube music (maybe not as easily anymore) and achieve a larger scale in a similar quality level.
now if hypothetically tidal had all the music of the world and was accessible this way, then it would be a comparable resource. insane regardless.
i completely understand the archive's decision on applying their own compression.
It was quality in technical quality of the audio in the files, but also in the organization and sourcing of the material, the QA-process of the encoding - down the the specific release the audio-file was from.
There was quantity, sure, but that was secondary to the quality. The quantity was just a side-effect of the place being known for quality, making it an attractive arena to participate in.
And it also had all the "weird"/non-standard things you don't find on mainstream streaming-services precisely because that is what independent curators are good at and often driven by.
This Anna's release... While in itself impressive in many ways does not compare to the things What.CD represented. It's almost the exact opposite:
- focus on most popular content - niche content (even by mainstream Spotify-standards) is not included
- quality is 160kbps ogg files, which is far from lossless, it's not tightly coupled to a release and even as so far the audio-grading goes, there's no transparent QA process for the content, nor is it available in audiophile fidelity.
This is definitely Apples vs Oranges.
A distributed ripping project to do that would be a fine thing.
They also removed a lot of discovery features - Playlist Radio - for example. And they still do have some version of it on the backend, but you have to go through some weird mechanisms to trigger it - like play the last song in playlist, wait till it ends (or rewind) and you get the playlist radio. But it's also a crippled version of it - prefers playing the exact same popular songs for some reason.
Then they released this DJ thing, which is laughably bad. No Spotify, I don't want someone talking to me with useless information in between songs. Who though that was a good idea? Who actually uses that?
There hasn't been a change in Spotify in last 7 years or so that wasn't negative.
There is contemporary lost media being created every day because of how we distribute things now. I think in some cases, the intent of the publisher was to literally destroy every copy of the information. I understand the legal arguments for this, but from a spiritual perspective, this is one of the most offensive things I can imagine. Intentionally destroying all copies of a creative work is simply evil. I don't care how you frame it.
Making media effectively lost is not much different in my mind. Is it available if it's sitting on a tape in an iron mountain bunker that no one will ever look at again?
But, more importantly, I cannot even say "good for you", because I don't actually think it is good for Anna's Archive. I wouldn't touch that thing, if I was them. Do we even have any solid alternatives for books, if Anna's Archive gets shot down, by the way? Don't recommend Amazon, please.