This is a really really fun sounding project - ironically, because there are no audio samples provided at all. I would have thought a music producer creating samples for music would naturally let you listen to what they were making.
I always roll my eyes when I see LLM weirdos talk about getting models to run on "old" hardware and finding out it's hardware that's still better than what most people have access to.
It doesn't make it any less impressive to those who know what hardware requirements for LLMs usually is/are but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse and yet are unable to upgrade because of said LLMs stealing all the hardware in the world all while RAM/memory/storage manufacturers manipulate the market(s) against them.
Articles like this are why I come back to HN. Interesting technically, kinda novel and fun. Got me thinking about datasets that may be sitting on old HDD, got TBs of old video and audio from projects of past. Blogs like this help point the way.. Now if only I had the time..
People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:
1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags
2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.
My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)
Slightly off-topic. Now that 1920s jazz music is falling into public domain, has anyone tried to reinvigorate the music using AI and generative adversarial approaches? Pre-1940s music didn't have high-fidelity sound, so the strong bass lines weren't captured. In theory, we could "downgrade" modern recordings to sound like 1920s recordings, then use adversarial techniques to train the machine on how to restore the antique recordings. Anyone know of any work being done in this area?
I could take my sequencer and crank the tempo up to a level on a Chopin etude that would smoke Yuja Wang too.
Who cares? The performance that is interesting is a human performance under these artistic constraints.
We didn't need transformers for algorithmic jazz or algorithmic composition in general.
It is also the bullshit of algorithmic Bach. Bach produced 1,100 works and most people haven't listened to even 1% of arguably the greatest artist who ever lived. What is the point of generating more?
I have three (and pray I do not come up with any more) needs for AI audio apps.
One, a ubiquitous restoration model. Find degraded copies of music in the wild, old YouTube's, transcodes, vinyl rips, bad masters, half destroyed tapes... Pair them with modern pristine lossless encodes of the same music, train. Then use that model on music we don't have pristine copies of.
The second is similar but more specific. There are so many stems floating around from popular music. My idea is to compare individual stems against the results from MVS/Spleeter(same song, same instrument). This would surely stand a chance of pushing that tech forward, so we can treat the FFT artefact heavy sound of new efforts.
Thirdly, from a creative point of view, I wanna do the equivalent of image to image on my tracks... But I actually want it to hallucinate in the manner of the early deep dream images, I want to be able to play with that space..
I can knock out musak to spek in minutes already, gen music is just reducing low effort to nearly no effort, preventing people with needs from networking with creators.... Uhh.. but I think that's a very general issue with Gen AI away from the corporate/entrepreneurial dev space
This is cool, I had some questions because I was trying to understand this more - not the diffusion model training part, because that I've actually seen before- but the actual way you bundled this into a web application and the choices made...
Did you publish the actual trained model anywhere? I see how in the code there is python for how individual samples can be generated, but the model checkpoint pulldown comes from a directory that... I don't see.
I then went through the code of how this runs on the web and- I'm not a web dev guy- so I'm pretty confused at all the bits bolted together to make this into a web app. It seems like there is a WASM bit that is compiled from a typical C++ audio plugin that is doing the stuff like conv reverb and limiting and distortion in the web app - all that is oldschool, non-generative AI, DSP being applied to the samples. Then the samples are just... a few default generated samples, to start- where are they pulled from, physically? And you have a login requirement to spool up the actual generative AI part to generate new samples to run into the DSP (because that needs a GPU on the backend to do, so, a login to help rate limit this)
How big is the actual generative model? Did you ever think about building the generation engine into the WASM bundle, using maybe WebGPU in the WASM to accelerate in a platform agnostic way, so that the entire app would run offline in someone's browser window?
I'm having fun just playing with the kick program without a login, which, again, am I right in saying in that mode there is no gen AI of sampled happening server side, it is just playing starting with some pre-made samples?
What about copyright? Can one take 13,000 copyrighted works, train the model and release it for free so that nobody ever buys any of those 13,000 samples? Can one (legally) convert all commercial sample libraries into a free neural network?
Couldnt get sound to work on iOS but back at my Desktop now and wow these examples/presets sound really cool, great job! (The examples in the write up sound very weird btw...). Try out the 'app' here > https://kick-with-reverb.vercel.app/
28 comments
[ 2.0 ms ] story [ 49.4 ms ] threadIt doesn't make it any less impressive to those who know what hardware requirements for LLMs usually is/are but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse and yet are unable to upgrade because of said LLMs stealing all the hardware in the world all while RAM/memory/storage manufacturers manipulate the market(s) against them.
That I didn't get from the text.
1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags
2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.
My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)
[0]: https://soniccharge.com/synplant
I could take my sequencer and crank the tempo up to a level on a Chopin etude that would smoke Yuja Wang too.
Who cares? The performance that is interesting is a human performance under these artistic constraints.
We didn't need transformers for algorithmic jazz or algorithmic composition in general.
It is also the bullshit of algorithmic Bach. Bach produced 1,100 works and most people haven't listened to even 1% of arguably the greatest artist who ever lived. What is the point of generating more?
One, a ubiquitous restoration model. Find degraded copies of music in the wild, old YouTube's, transcodes, vinyl rips, bad masters, half destroyed tapes... Pair them with modern pristine lossless encodes of the same music, train. Then use that model on music we don't have pristine copies of.
The second is similar but more specific. There are so many stems floating around from popular music. My idea is to compare individual stems against the results from MVS/Spleeter(same song, same instrument). This would surely stand a chance of pushing that tech forward, so we can treat the FFT artefact heavy sound of new efforts.
Thirdly, from a creative point of view, I wanna do the equivalent of image to image on my tracks... But I actually want it to hallucinate in the manner of the early deep dream images, I want to be able to play with that space..
I can knock out musak to spek in minutes already, gen music is just reducing low effort to nearly no effort, preventing people with needs from networking with creators.... Uhh.. but I think that's a very general issue with Gen AI away from the corporate/entrepreneurial dev space
I love these one-liner explanations - the absolute minimum information one needs to move forward. Great writing.
Did you publish the actual trained model anywhere? I see how in the code there is python for how individual samples can be generated, but the model checkpoint pulldown comes from a directory that... I don't see.
I then went through the code of how this runs on the web and- I'm not a web dev guy- so I'm pretty confused at all the bits bolted together to make this into a web app. It seems like there is a WASM bit that is compiled from a typical C++ audio plugin that is doing the stuff like conv reverb and limiting and distortion in the web app - all that is oldschool, non-generative AI, DSP being applied to the samples. Then the samples are just... a few default generated samples, to start- where are they pulled from, physically? And you have a login requirement to spool up the actual generative AI part to generate new samples to run into the DSP (because that needs a GPU on the backend to do, so, a login to help rate limit this)
How big is the actual generative model? Did you ever think about building the generation engine into the WASM bundle, using maybe WebGPU in the WASM to accelerate in a platform agnostic way, so that the entire app would run offline in someone's browser window?
I'm having fun just playing with the kick program without a login, which, again, am I right in saying in that mode there is no gen AI of sampled happening server side, it is just playing starting with some pre-made samples?