This is so good that I wondered if it's fake. Really impressive results from generated spectrographs! Also really interesting that it's not exactly trained on the audio files themselves - wonder if the usual copyright-based objections wild even apply here.
regarding those usual objections, i'd argue that a spectrograph representation of a given piece of audio is just a different (lossy) encoding of the same content/information, so any hypothetical objections would still apply here.
You would be absolutely correct. the lossiness is in the resolution of the image (512x512 is pretty terrible) but given enough image resolution it's just an FFT transform, and the only reason that stuff falls short is because people don't give it, in turn, enough resolution. If you did wild overkill of the resolution of an FFT transform you could do anything you wanted with no loss of tone quality. If you turned that to visual images and did diffusion with it you could do AI diffusion at convincing audio quality.
In theory the tone quality is not an objection here. When it sounds bad it's because it's 512x512, because the FFT resolution isn't up to the task, etc. People cling to very inadequate audio standards for digital processing, but you don't have to.
Rearrange that trip through latent space a little, jumping back and forth through different stages of the interpolation in a pattern resembling those customary chorus/verse things and you've got a hit. And you could reuse the exact same rearrange recipe for just about any interpolation between prompt pairs.
Plenty of times this has been called before, and it's certainly possible that this wont be the last time this is called, but allow me to declare this the end of the bedroom producer (stage performers remain unaffected). And no two elevators will ever sound the same again, on any day.
Producing images of spectrograms is a genius idea. Great implementation!
A couple of ideas that come to mind:
- I wonder if you could separate the audio tracks of each instrument, generate separately, and then combine them. This could give more control over the generation. Alignment might be tough, though.
- If you could at least separate vocals and instrumentals, you could train a separate model for vocals (LLM for text, then text to speech, maybe). The current implementation doesn't seem to handle vocals as well as TTS models.
great stuff, while it comes with the usual smeary iFFT artifacts that AI-generated sound tends to have the results are surprisingly good.
i especially love the nonsense vocals it generates in the last example, which remind me of what singing along to foreign songs felt like in my childhood.
impressive stuff. reminds me of when ppl started using image classifier networks on spectrograms in order to classify audio. i would not have thought to apply a similar concept for generative models, but it seems obvious in hindsight.
Nice reference! I had never seen that site before, but those albums had a significant impact on my musical journey.
There was a purple victorian house in Colorado Springs where the living room was converted into a record and cd store called Life By Design. I picked up these albums and a ton of other obscure music there. I was so happy to not have to drive all the way up to Wax Trax in Denver to be able to discover new artists.
Fun! I tried something similar with DCGAN when it first came out, but that didn't exactly make nice noises. The conversion to and from Mel spectrograms was lossy (to put it mildly), and DCGAN, while impressive in its day, is nothing like the stuff we have today.
Interesting that it gets so good results with just fine tuning the regular SD model. I assume most of the images it's trained on are useless for learning how to generate Mel spectrograms from text, so a model trained from scratch could potentially do even better.
There's still the issue of reconstructing sound from the spectrograms. I bet it's responsible for the somewhat tinny sound we get from this otherwise very cool demo.
This is amazing! Would it be possible to use it to modify this interpolate between two existing songs (i.e. generate spectrograms from audio and transition between them)?
This really is unreasonably effective. Spectrograms are a lot less forgiving of minor errors than a painting. Move a brush stroke up or down a few pixels, you probably won't notice. Move a spectral element up or down a bit and you have a completely different sound. I don't understand how this can possibly be precise enough to generate anything close to a cohesive output.
You can also add another neural-network to "smooth" the spectrogram, increase the resolution and remove artefacts, just like they do for image generation.
Wasn't this Fraunhofer's big insight that led to the development of MP3? Human perception actually is pretty forgiving of perturbations in the Fourier domain.
In very limited situations. You can move a frequency around (or drop it entirely) if it's being masked by a nearby loud frequency. Otherwise, you would be amazed at the sensitivity of pitch perception.
The easy example of this is playing a slightly out of tune guitar, or a mandolin where the strings in the course aren't matched in pitch perfectly. You can hear it, and it's just a few cents off.
You probably mean Karlheinz Brandenburg, the developer of MP3, who worked on psychoacoustics. Not completely off though, as he did the research at a Fraunhofer research institute, which takes its name from Joseph von Fraunhofer, the inventor of the spectroscope.
Fair enough. But for me, when talking about `having an insight`, I don't imagine a non-human entity doing that. And to be pedantic (talking about Germans doing research, I hope everyone would expect me to be), the institute is called Fraunhofer IIS. `Fraunhofer` would colloquially refer to the society, which is an organization with 76 institutes total. Although, of course, the society will also claim the work...
It's an interesting question, one I hadn't thought of before. But in common language, it sometimes makes sense to credit the institution, others just the individuals. I think may be more based around how much the institution collectively presents itself as the author and speaks on behalf of the project versus the individuals involved. Here is my own general intuition for a few contrasting cases:
Random forests: Ko and Breiman's, not really Bell Labs and UC-Berkeley
Transistors: Bardeen, Brattain, and Shockley, not really Bell Labs (thank the Nobel Prize for that)
UNIX: Primarily Bell Labs, but also Ken Thompson and Dennis Richie (this is a hard one)
GPT-n: OpenAI, not really any individual, and I can't seem to even recall any named individual from memory
Author here: We were blown away too. This project started with a question in our minds about whether it was even possible for the stable diffusion model architecture to output something with the level of fidelity needed for the resulting audio to sound reasonable.
Excellent work! Singing would be amazing - karaoke can finally sound good :p
Have you released a tool for volumetric capture? I'm applying this to LED lighting fixture setup for tv/film/live shows and 3D positioning is the last step to fully automated configuration.
My goal is real-time sync between 3D model and real world.
I think we're at the point, with these AI generative model thingies, where the practitioners are mesmerized by the mechatronic aspect like a clock maker who wants to recreate the world with gears, so they make a mechanized puppet or diorama and revel in their ingenuity.
This is a genius idea. Using an already-existing and well-performing image model, and just encoding input/output as a spectrogram... It's elegant, it's obvious in retrospective, it's just pure genius.
I can't wait to hear some serious AI music-making a few years from now.
Makes me wonder if we will see a generalization of this idea. Just like in a CPU 90%+ of want you want to do can be modeled with very few instructions (mov, add, jmp..) we could see a set of very refined models (Stable difussion, GPT, etc) and all of their abstractions on top (ChatGPT, Rifussion, etc).
and you ask stable diffusion to generate piet code for a slightly better version of stable diffusion (or chatGPT) ...which then you can further use to generate a better version, and so on. Singularity here we come!
For what is worth, people were trying the same thing with GANs (I also played with doing it with stylegan a bit) but the results weren't as good.
The amazing thing is that the current diffusion models are so good that the spectograms are actually reasonable enough despite the small room for error.
> I can't wait to hear some serious AI music-making a few years from now.
I think this will be particularly useful for musical compositions in movies and film, where the producer can "instruct" the AI about what to play, when, and how to transition so that the music matches the scene progression.
Not only that but sampling. I'd say there's at least one sample from something in most modern music. This can essentially create "sounds" that you're looking for as an artist. I need a sort of high pitched drone here... Rather than dig through sample libraries you just generate a few dozen results from a diffusion model with some varying inputs and you'd have a small sample set on the exact thing you're looking for. There's already so much processing of samples after the fact, the actual quality or resolution of the sample is inconsequential. In a lot of music, you're just going after the texture and tonality and timbre of something... This can be seen in some Hans Zimmer videos of how he slows down certain sounds massively to arrive at new sounds... or in granular synthesis... This is going to open up a lot of cool new doors.
I suspect that if you had tried this with previous image models the results would have been terrible. This only works since image models are so good now.
I'm super excited about the Audio AI space, as it seems permanently a few years behind image stuff - so I think we're going to see a lot more of this.
If you're interested, the idea of applying Image processing techniques to Spectrograms of audio is explored in brief in the first lesson of one of the most recommended AI courses on HN: Practical Deep Learning for Coders https://youtu.be/8SF_h3xF3cE?t=1632
I dabble in music production and know some of the people in the "Lofi" world, so I know for a fact that this is not true. It's just a formulaic sub-genre where people are trying to make similar instrumentals with the same vibe. It would be jarring to listen to a playlist while studying and each song had wildly different tempos, instruments, etc.
Also, the music doesn't sound "Lofi" because it's generated by algorithms. A lot of hard work and software goes into taking a clean, pitch-perfect digital signal and making it sound like something playing on a record player from the 70s.
This idea is presented by Jeremy Howard on literally their first Deep Learning for Coders class (most recent edition). A student wanted to classify sounds, but only knew how to do vision, so they converted sounds to spectrograms, fine tuned the model on the labelled spectra, and the classification worked pretty well on test data.
That of course does not take the merit away from the Riffusion authors though.
The idea of connecting CV to audio via spectrograms pre dates Jeremy Howard's course by quite a bit. That's not really the interesting part here though. The fact that a simple extension of an image generation pipeline produces such impressive results with generative audio is what is interesting. It really emphasizes how useful the idea of stable diffusion is.
The idea to apply computer vision algorithms to spectrograms is not new. I don't know who first came up with it, but I first came across it about a decade ago.
There were a number of systems designed for composers in the 90s (also continuing through to today) designed for the workflow of converting a sound to a spectrogram, doing visual processing on the image, and then re-synthesizing the sound from the altered spectrogram. Many were inspired by Xenakis' UPIC system which was designed around the second half of this workflow: you'd draw the spectrogram with a pen and then synthesize it.
Edit: my favorite of all these systems was Chris Penrose's HyperUPIC which provided a lot of freedom in configuring how the analysis and synthesis steps worked.
As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad.
Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to be poorly suited for?
As someone who loves making music and loves listening to music (regardless of its origins, in my case), it doesn't make me that sad. Sure, at first, I had an uncomfortable feeling that AI could make this sacred magic thing that only I and other fellow humans know how to do... But then I realized same thing is happening with visual art, so I applied the same counterarguments that've been cooking in my head.
I think that kind of attitude is defeatist - it's implying that humans will be stopped from making music if AI learns how to do it too. I don't think that will happen. Humans will continue making music, as they always have. When Kraftwerk started using computers to make music back in the 70s, people were also scared of what that will do to musicians. To be fair, live music has died out a bit (in a sense that there aren't that many god-on-earth-level rockstars), but it's still out there, people are performing, and others who want to listen can go and listen.
Maybe consumers will start consuming more and more AI music, instead of human music [0], but the worst thing that can happen is that music will no longer be a profitable activity. But then again, today's music industry already has some elements of the automation - washed-out rhythms, sexual thematics over and over again, re-hashing same old songs in different packages... So nothing's gonna change in the grand scheme of things.
> but the worst thing that can happen is that music will no longer be a profitable activity.
For me, the worst that could happen is that people spend so much time listening to AI generated music, that human musicians can no longer find audiences to connect to. It's not just about economics (though that's also huge). It's the psychological cost of all of us spending greater and greater fractions of our lives connected to machines and not other people.
Music was always about people. Even today, as most people listen to the mass-produced run-of-the-mill muzak, there is still a significant audience that seeks the "human element" for the sake of itself.
Black metal community, for example, has always rejected all forms of "automation" and considers it not kvlt - rawness is a sought-after quality, defined as having people performing as close to the recording equipment as possible.
There's also a rapper named Bones (Elmo O'Connor) who's never signed a contract with a label, does only music he wants to do, releases a couple albums every year. There's something about his approach that makes his music sound very organic and honest. I listen to him more than I listen to any mass produced rapper.
So in conclusion, music was always about people. Unless AI reaches AGI level, I don't think it will ever impact music enough to kill all audience.
I agree with this so much that I’d take off the comment about AGI: I think that even if/when AGI lands there will always be audiences that seek raw, organic, and honest artistic expression from other humans.
I would rather live in a world where humans get to do things they love because they can (and not because they have to earn their bread), and machines get to do basically everything that needs to be done but no human is willing to do it.
Advancing AI capabilities in no way detracts from this. You talk about humans being "relegated to the remaining tasks" - but that's a consequence of our socioeconomic system, not of our technology.
I'd rather live in the world where humans do things that are actually unique and interesting, and aren't essentially being artificially propped up by limiting competition.
I don't see this as a threat to human ingenuity in the slightest.
I play the piano (badly). There are many other people who can play much better than I. There are simple computer programs which can play better. It doesn't stop me from enjoying it or playing it. Computers have been beating people at Chess for years yet you still see people everywhere enjoying the game. At some point computers will be better than humans at absolutely everything but it shouldn't stop you as a human from enjoying anything.
Sure, but a large part of enjoyment in creativity for many is the joy of sharing it with an audience. To the degree that people are spending their available attention on AI-generated content, they have less time and attention available to spend listening to and watching art created by humans.
Musicians already make much (most?) of their money via gigs and I don't think going to watch an AI play at a gig will be all too common. I think we'll be fine. Might have to adapt though.
Congratulations this is an amazing application of technology and truly innovative. This could be leveraged by a wide range of applications that I hope you'll capitalize on.
I agree but it's better than listening to Ed Sheeran
Edit: To be honest, I find something like 'Band In A Box' to be more impressive and actually useful, I don't understand how I would ever use this or listen to this. To me, it's further proof that Stable Diffusion really just doesn't work that well
Very cool! I was wondering why there wasnt any music diffusion apps out there, it seems more useful because music has stricter copyright and all content creators need some background music ...
"We should place a tax on all copyright lawyers and use it to fund GPU manufacturing and AI development. At your next stump speech, mention how the entertainment industry is stealing jobs from construction workers. Your corruption charges won't matter because voters only care about corruption when it's not in their favor."
Although I do wonder how much an earlier technology, audio reproduction, contributed to that. My grandmother worked for a time as a piano player as part of a nightclub orchestra. It was a stable job back then. I have to wonder how many musician jobs were killed off by the jukebox and related technologies.
If a hack based on an image generator already has promising results for music generation, then imagine what will happen if something dedicated to music is built from the ground up.
What is with the hyperbole in this thread? This stuff sounds like incoherent noise. It is noticeably worse than AI audio stuff I heard 5 years ago. What is going on with the responses here?
Usage of an image generator to produce passable music fragments, even if they sound a bit distorted, is very surprising. That type of novelty is why we come here.
I assume the stuff from 5 years ago was essentially spitting out a midi output which would be fed in to a traditional tool to play samples. So it's going to sound a lot sharper while being a lot less sophisticated. The real breakthrough here is this is generating everything from scratch and it still resembles the prompt.
One of the automated prompts was "Eminem anger rap", I'm confident if you had showed me the audio without the prompt I could identify which artist it sounded like.
And this is just a basic first attempt at reusing a tool not even designed for audio. I can only imagine how powerful it could be after some trivial revisions like using GPT-3 to generate coherent lyrics.
I feel exactly the opposite way, but I suppose everyone has a different ear and taste. I think a good 3-4% of what this produces sounds damn amazing and beautiful. I've been vibing to it a lot. Fantastic stuff! There is also the feeling of shock and awe like with ChatGPT where you give it a prompt about a niche thing you think it will definitely not understand and it turns out it understands it shockingly well. As an example I just gave it a prompt "Avril 4th" and the result literally gave me chills.
Have you tried AI asset generators? They are working extremely good. Just yesterday a friend of mine has shown me the progress they made in their game. It is incredible.
Designers are 100% loosing their job over this.
I'm a professional game developer and excited AI enthusiast.
While I've seen a lot of cool stuff which helps generation for hobby projects or smaller indie games it's nowhere near the quality and consistency needed to come close to the work of a skilled human artist at a larger studio.
Yes, and I studied Game Development in Germany for 3 years in Düsseldorf, got third at the national gameforge newcomer award with my team "Northlight Games" and still have many connections to the people in the business (if this somehow matters). The quality for 2d assets is on very professional level and already replaced jobs in projects I know.
To give you an example join the public discord of https://www.scenario.gg and check out results. Come back and tell me those aren't on a professional level.
I am not saying that designers won't be needed anymore but AI is definitely able to replace jobs and speed up progress in game development.
Some of this is really cool! The 20 step interpolations are very special, because they're concepts that are distinct and novel.
It absolutely sucks at cymbals, though. Everything sounds like realaudio :) composition's lacking, too. It's loop-y.
Set this up to make AI dubtechno or trip-hop. It likes bass and indistinctness and hypnotic repetitiveness. Might also be good at weird atonal stuff, because it doesn't inherently have any notion of what a key or mode is?
As a human musician and producer I'm super interested in the kinds of clarity and sonority we used to get out of classic albums (which the industry has kinda drifted away from for decades) so the way for this to take over for ME would involve a hell of a lot more resolution of the FFT imagery, especially in the highs, plus some way to also do another AI-ification of what different parts of the song exist (like a further layer but it controls abrupt switches of prompt)
It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do indistinct. It's realaudio grade, it needs to be more like 128kbps mp3 grade.
Well no wonder, it has absolutely no concept of composition beyond a single 5s loop, if I understand correctly.
> It absolutely sucks at cymbals, though. Everything sounds like realaudio :)
> It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do indistinct. It's realaudio grade, it needs to be more like 128kbps mp3 grade.
I haven't sat down yet to calculate it, but is the output of SD at 512*512px at 24bit enough to generate audio CD quality in theory?
And I suspect this will always have phase smearing, because it's not doing any kind of source separation or individual synthesis. It's effectively a form of frequency domain data compression, so it's always going to be lossy.
It's more like a sophisticated timbral morph, done on a complete short loop instead of an individual line.
It would sound better with a much higher data density. CD quality would be 220500 samples for each five second loop. Realtime FFTs with that resolution aren't practical on the current generation of hardware, but they could be done in non-realtime. But there will always be the issue of timbres being distorted because outside of a certain level of familiarity and expectation our brains start hearing gargly disconnected overtones instead of coherent sound objects.
What this is not doing is extracting or understanding musical semantics and reassembling them in interesting ways. The harmonies in some of these clips are pretty weird and dissonant, and not what you'd get from a human writing accessible music. This matters because outside of TikTok music isn't about 5s loops, and longer structures aren't so amenable to this kind of approach.
This won't be a problem for some applications, but it's a long way short of the musical equivalent of a MidJourney image.
Generally we're a lot more tolerant of visual "bugs" than musical ones.
I think an approach like this could generate interesting sounds we as humans would never think of. Or meshing two sounds in ways we could barely imagine or implement.
But of course something like this, which only thinks in 5s clips can not generate a larger structure, like even a simple song. Maybe another algorithm could seed the notes and an algorithm like this generates the sounds via img2img.
>and not what you'd get from a human writing accessible music
The timbral qualities of the posted samples remind me of some of the stuff I heard from Aphex Twin, like Alberto Balsalm. Not accessible by a long shot but definitely human
This is really cool but can someone tell me why we are automating art? Who asked for this? The future seems depressing when I look at all this AI generated art.
You can't automate a live performance or an oil painting with AI in this way. This isn't going to replace musicians and artists. If anything, I think a preponderance of AI art would make people appreciate the real stuff more.
As to why, music is fun to create, and this is just a tool.
Everyone asked for this, including artists. If you make a living off of making art, having the best tools to help you do that is a constant, and the tools are finally starting to get properly good. Will "the job" change because of the tools? Of course. Will the nature of what it means for something to be art change? Also of course. Art isn't some static, untouchable thing. It changes as humanity does.
Because tons of people want to make art, and a lot of art currently requires years of training to make anything close to "good". Making art more accessible to create is a boon to everyone who's dreamed of being able to make their own paintings and music, but doesn't have the skills required.
Actually I agree with you, but HN is not really a place where you will find artists defending themselves. However you will find alot of people defending the automation of art. Generative art has it's place. But ultimately until humans are extinct, human generated art is the only thing which really represents the species. Everything else is an advanced form of puppetry or mimicry.
I would say it's not "generated," but interpolated...
It doesn't make anything new or fresh. It doesn't pull any real-life emotions or experiences into a synthesis that a person can relate to. It's more like asking a teenaged comedian to imitate numerous impressions of music styles. e.g. in Clerks when the Russian guy does "metal": https://youtu.be/7gFoHkkCaRE?t=55
Of course the modern conception of music in the West is as an accompaniment to other, mostly drudging, activities, as opposed to something to be paid singularly attention. Therefore, there are many "valuable"(*) occasions to produce "impressions" of music. E.g. in advertisements and social media flexes where identity and attitude are the purpose of music. For these, a shallow interpretation or reflection of loosely amalgamated sound clips will suffice. But we don't just attend concerts or focus sustained energy on sonic impressions. We listen to lyrics and give over our consciousness to composed works because we want to find secrets others give away in dealing with this crazy thing called life- ideas to succeed, admissions of failure, and what the expected emotional arcs of these trajectories looks like. This lofty goal is to date not within the scope of AI stunts.
As Solzheinetysn said, "Too much art is like candy and not bread."
We're not automating art, we're creating tools that make it easier for humans to create art. These are nothing more than new and exciting tools. The cream will still rise to the top, same as it ever was.
The vocals in these tracks are so interesting. They sound like vocals, with the right tone, phonemes. and structure for the different styles and languages but no meaning.
480 comments
[ 0.21 ms ] story [ 315 ms ] threadIn theory the tone quality is not an objection here. When it sounds bad it's because it's 512x512, because the FFT resolution isn't up to the task, etc. People cling to very inadequate audio standards for digital processing, but you don't have to.
Might be a traffic thing?
Edit: Works now. A bit laggy but it works. Brilliant!
{"data":{"success":true,"worklet_output":{"error":"Model version 5qekv1q is not healthy"},"latency_ms":530}}
https://www.youtube.com/watch?v=crcqADcAusg
Plenty of times this has been called before, and it's certainly possible that this wont be the last time this is called, but allow me to declare this the end of the bedroom producer (stage performers remain unaffected). And no two elevators will ever sound the same again, on any day.
A couple of ideas that come to mind:
- I wonder if you could separate the audio tracks of each instrument, generate separately, and then combine them. This could give more control over the generation. Alignment might be tough, though.
- If you could at least separate vocals and instrumentals, you could train a separate model for vocals (LLM for text, then text to speech, maybe). The current implementation doesn't seem to handle vocals as well as TTS models.
There was a purple victorian house in Colorado Springs where the living room was converted into a record and cd store called Life By Design. I picked up these albums and a ton of other obscure music there. I was so happy to not have to drive all the way up to Wax Trax in Denver to be able to discover new artists.
Interesting that it gets so good results with just fine tuning the regular SD model. I assume most of the images it's trained on are useless for learning how to generate Mel spectrograms from text, so a model trained from scratch could potentially do even better.
There's still the issue of reconstructing sound from the spectrograms. I bet it's responsible for the somewhat tinny sound we get from this otherwise very cool demo.
Absolutely blows my mind.
Random forests: Ko and Breiman's, not really Bell Labs and UC-Berkeley
Transistors: Bardeen, Brattain, and Shockley, not really Bell Labs (thank the Nobel Prize for that)
UNIX: Primarily Bell Labs, but also Ken Thompson and Dennis Richie (this is a hard one)
GPT-n: OpenAI, not really any individual, and I can't seem to even recall any named individual from memory
[1] https://www.respeecher.com/
I had a look around several months ago, and it seems like everything is locked behind SaaS APIs.
Steve Blum: https://fakeyou.com/tts/result/TR:xmjjq9ty0hnsyjrjnw806k6rnp...
Furiously working on voice-to-voice (web, real time, and singing!) Should be out the door tomorrow!
Have you released a tool for volumetric capture? I'm applying this to LED lighting fixture setup for tv/film/live shows and 3D positioning is the last step to fully automated configuration.
My goal is real-time sync between 3D model and real world.
How do you think human endeavours progress other than by small steps?
I can't wait to hear some serious AI music-making a few years from now.
https://www.dangermouse.net/esoteric/piet.html
The amazing thing is that the current diffusion models are so good that the spectograms are actually reasonable enough despite the small room for error.
I think this will be particularly useful for musical compositions in movies and film, where the producer can "instruct" the AI about what to play, when, and how to transition so that the music matches the scene progression.
If you're interested, the idea of applying Image processing techniques to Spectrograms of audio is explored in brief in the first lesson of one of the most recommended AI courses on HN: Practical Deep Learning for Coders https://youtu.be/8SF_h3xF3cE?t=1632
Also, the music doesn't sound "Lofi" because it's generated by algorithms. A lot of hard work and software goes into taking a clean, pitch-perfect digital signal and making it sound like something playing on a record player from the 70s.
edit: added a bit more to the thought
I just ran a quick Google Scholar search, and the first result is https://ieeexplore.ieee.org/abstract/document/5672395
This is from 2010. I didn't go looking, but it wouldn't surprise me if the idea is older than that.
https://en.wikipedia.org/wiki/UPIC
Edit: my favorite of all these systems was Chris Penrose's HyperUPIC which provided a lot of freedom in configuring how the analysis and synthesis steps worked.
Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to be poorly suited for?
I think that kind of attitude is defeatist - it's implying that humans will be stopped from making music if AI learns how to do it too. I don't think that will happen. Humans will continue making music, as they always have. When Kraftwerk started using computers to make music back in the 70s, people were also scared of what that will do to musicians. To be fair, live music has died out a bit (in a sense that there aren't that many god-on-earth-level rockstars), but it's still out there, people are performing, and others who want to listen can go and listen.
Maybe consumers will start consuming more and more AI music, instead of human music [0], but the worst thing that can happen is that music will no longer be a profitable activity. But then again, today's music industry already has some elements of the automation - washed-out rhythms, sexual thematics over and over again, re-hashing same old songs in different packages... So nothing's gonna change in the grand scheme of things.
[0] https://www.youtube.com/watch?v=S1jWdeRKvvk
For me, the worst that could happen is that people spend so much time listening to AI generated music, that human musicians can no longer find audiences to connect to. It's not just about economics (though that's also huge). It's the psychological cost of all of us spending greater and greater fractions of our lives connected to machines and not other people.
Black metal community, for example, has always rejected all forms of "automation" and considers it not kvlt - rawness is a sought-after quality, defined as having people performing as close to the recording equipment as possible.
There's also a rapper named Bones (Elmo O'Connor) who's never signed a contract with a label, does only music he wants to do, releases a couple albums every year. There's something about his approach that makes his music sound very organic and honest. I listen to him more than I listen to any mass produced rapper.
So in conclusion, music was always about people. Unless AI reaches AGI level, I don't think it will ever impact music enough to kill all audience.
Advancing AI capabilities in no way detracts from this. You talk about humans being "relegated to the remaining tasks" - but that's a consequence of our socioeconomic system, not of our technology.
Those two are profoundly intertwined. Our tech affects our socioeconomic systems and vice versa.
I don't see this as a threat to human ingenuity in the slightest.
Edit: To be honest, I find something like 'Band In A Box' to be more impressive and actually useful, I don't understand how I would ever use this or listen to this. To me, it's further proof that Stable Diffusion really just doesn't work that well
Who else will AI make looking for a new job?
GPT-3, what policy should we apply to increase tax revenue by 5% given these constraints?
GPT-3, please tell me some populist thing to say to win the next election, or how should I deflect these corruption charges.
One of the automated prompts was "Eminem anger rap", I'm confident if you had showed me the audio without the prompt I could identify which artist it sounded like.
And this is just a basic first attempt at reusing a tool not even designed for audio. I can only imagine how powerful it could be after some trivial revisions like using GPT-3 to generate coherent lyrics.
While I've seen a lot of cool stuff which helps generation for hobby projects or smaller indie games it's nowhere near the quality and consistency needed to come close to the work of a skilled human artist at a larger studio.
To give you an example join the public discord of https://www.scenario.gg and check out results. Come back and tell me those aren't on a professional level.
I am not saying that designers won't be needed anymore but AI is definitely able to replace jobs and speed up progress in game development.
It absolutely sucks at cymbals, though. Everything sounds like realaudio :) composition's lacking, too. It's loop-y.
Set this up to make AI dubtechno or trip-hop. It likes bass and indistinctness and hypnotic repetitiveness. Might also be good at weird atonal stuff, because it doesn't inherently have any notion of what a key or mode is?
As a human musician and producer I'm super interested in the kinds of clarity and sonority we used to get out of classic albums (which the industry has kinda drifted away from for decades) so the way for this to take over for ME would involve a hell of a lot more resolution of the FFT imagery, especially in the highs, plus some way to also do another AI-ification of what different parts of the song exist (like a further layer but it controls abrupt switches of prompt)
It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do indistinct. It's realaudio grade, it needs to be more like 128kbps mp3 grade.
Well no wonder, it has absolutely no concept of composition beyond a single 5s loop, if I understand correctly.
> It absolutely sucks at cymbals, though. Everything sounds like realaudio :)
> It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do indistinct. It's realaudio grade, it needs to be more like 128kbps mp3 grade.
I haven't sat down yet to calculate it, but is the output of SD at 512*512px at 24bit enough to generate audio CD quality in theory?
And I suspect this will always have phase smearing, because it's not doing any kind of source separation or individual synthesis. It's effectively a form of frequency domain data compression, so it's always going to be lossy.
It's more like a sophisticated timbral morph, done on a complete short loop instead of an individual line.
It would sound better with a much higher data density. CD quality would be 220500 samples for each five second loop. Realtime FFTs with that resolution aren't practical on the current generation of hardware, but they could be done in non-realtime. But there will always be the issue of timbres being distorted because outside of a certain level of familiarity and expectation our brains start hearing gargly disconnected overtones instead of coherent sound objects.
What this is not doing is extracting or understanding musical semantics and reassembling them in interesting ways. The harmonies in some of these clips are pretty weird and dissonant, and not what you'd get from a human writing accessible music. This matters because outside of TikTok music isn't about 5s loops, and longer structures aren't so amenable to this kind of approach.
This won't be a problem for some applications, but it's a long way short of the musical equivalent of a MidJourney image.
Generally we're a lot more tolerant of visual "bugs" than musical ones.
But of course something like this, which only thinks in 5s clips can not generate a larger structure, like even a simple song. Maybe another algorithm could seed the notes and an algorithm like this generates the sounds via img2img.
The timbral qualities of the posted samples remind me of some of the stuff I heard from Aphex Twin, like Alberto Balsalm. Not accessible by a long shot but definitely human
As to why, music is fun to create, and this is just a tool.
You'd have to combine it with these guys https://www.youtube.com/watch?v=WqE9zIp0Muk
I did.
> The future seems depressing when I look at all this AI generated art.
You should talk about your concerns with an AI psychotherapist.
It doesn't make anything new or fresh. It doesn't pull any real-life emotions or experiences into a synthesis that a person can relate to. It's more like asking a teenaged comedian to imitate numerous impressions of music styles. e.g. in Clerks when the Russian guy does "metal": https://youtu.be/7gFoHkkCaRE?t=55
Of course the modern conception of music in the West is as an accompaniment to other, mostly drudging, activities, as opposed to something to be paid singularly attention. Therefore, there are many "valuable"(*) occasions to produce "impressions" of music. E.g. in advertisements and social media flexes where identity and attitude are the purpose of music. For these, a shallow interpretation or reflection of loosely amalgamated sound clips will suffice. But we don't just attend concerts or focus sustained energy on sonic impressions. We listen to lyrics and give over our consciousness to composed works because we want to find secrets others give away in dealing with this crazy thing called life- ideas to succeed, admissions of failure, and what the expected emotional arcs of these trajectories looks like. This lofty goal is to date not within the scope of AI stunts.
As Solzheinetysn said, "Too much art is like candy and not bread."
Reminds me of the soundtrack to Nier Automata which did a similar thing: https://youtu.be/8jpJM6nc6fE