HN Explorer

Show HN: Cloning a musical instrument from 16 seconds of audio (erlj.notion.site)

185 points by abdljasser2 ↗ HN
In 2020, Magenta released DDSP [1], a machine learning algorithm / python library which made it possible to generate good sounding instrument synthesizers from about 6-10 minutes of data. While working with DDSP for a project, we realised how it was actually quite hard to find 6-10 minute of clean recordings of monophonic instruments.

In this project, we have combined the DDSP architecture with a domain adaptation technique from speech synthesis [2]. This domain adaptation technique works by pre-training our model on many different recordings from the Solos dataset [3] first and then fine-tuning parts of the model to the new recording. This allows us to produce decent sounding instrument synthesisers from as little as 16 seconds of target audio instead of 6-10 minutes.

[1] https://arxiv.org/abs/2001.04643

[2] https://arxiv.org/abs/1802.06006

[3] https://arxiv.org/abs/2006.07931

We hope to publish a paper on the topic soon.

74 comments

[ 3.0 ms ] story [ 125 ms ] thread
[deleted]
(comment deleted)
smaddox
Very cool. I wonder if you could turn something like this into the ultimate keyboard synth. That would be one hell of a product.
abdljasser2
Copy pasting my a reply to a similar question

Short answer is yes! Previous work has shown that we can obtain very good results from controlling DDSP models from midi input. The solutions I am familiar with employ a two stage approach where the first stage takes midi and turns it into control signals (pitch & loudness contours etc..) and the second stage turns the controls signals into audio (like the particular model I discuss in the blog post)[1][2][3]. I actually think that the first stage could also benefit from the transfer learning techniques we discuss in the blogpost. In terms of actually releasing a MIDI playable VST plugin I believe that Magenta have something like it in the works[4]. I hope that it will come with some ability for users to quickly create their own instruments, presumably using a transfer learning technique similar to the one we have presented.

Real-time rendering poses multiple challenges. For one, some instrument sounds occur before a note properly onsets (for example the sound of the fingers pressing the keys of a saxophone occurs before the first note of the piece). Secondly, the research models are quite heavy and considerably more compute intensive than a standard VST instrument which poses a problem if you want to use it inside a DAW. I think this latter problem can be solved with some clever engineering and the general trend of hardware being more and more accommodating to machine learning applications.

[1] https://erl-j.github.io/controlsynthesis/#/ (Our previous work) [2] https://rodrigo-castellon.github.io/midi2params/ (Focuses on realtime rendering) [3] https://arxiv.org/abs/2112.09312 (Magenta's recent paper on the subject)

skybrian
Very interesting! Is there a preprint or demo that you forgot to link to? Will you be releasing source or data?

I'm wondering what the hardware requirements would be for real-time synthesis. I work on a musical instrument project as a hobby and would like a good accordion sound.

michae2
Wow, this seems like it could be bigger than Auto-Tune. I wonder if we'll reach a point where artists license a DDSP model of their instrument or voice, rather than perform directly.
birdyrooster
Well, I suppose a form of this technically has already happened. Sakiko Fujita is the voice actor who was employed to create the samples for the Vocaloid Hatsune Miku. The copyright holder currently is SEGA.
ajross
While it's doing a great job emulating the timbre of the notes, what's interesting to me is what it's not doing. Play those top two passages to anyone who's spent time around sax players, and they'll instantly tell you which one was the "fake".

Real instruments have mechanical behavior this isn't going to see absent some kind of modeling. Different notes on a sax (to take this example) are actuated by different fingers and different valves and have different (often multiple) embouchures, and both interact with the transitions between pairs of notes (and with the dynamics with which they're played). All that complexity is absolutely hearable in the transitions between notes, and the ML layers aren't going to have the ability to pick it up absent a much larger training set.

Basically: 16 seconds of audio is enough to get you the frequency spectrum of the notes, which you can do with 3-4 lines of synthesis code in an imperative regime. It's very much the "easy part" of instrument synthesis.

davio
I played sax through college and the second passage sounds legit to me.

Couple of interesting things: - You can hear the keys and pads hitting and clicking - Vibrato on the last note is very realistic - Can hear the air and "spit" - The timing seems human, especially as it slows down a little on the lower notes which take more air and usually involve more awkward fingering with the pinky fingers. Definitely sounds like a non-professional, probably how an average high school player would play the passage.

It sounds like what a player would hear versus a studio recording. Reminds me of being in a really reflective & tiny practice room.

LegitShady
ir you read down the article the 16 seconds is just the input to a ml model. the actual training set is fifty something minutes.
scrozier
But if one is trying to create a great sounding instrument synthesizer, it is quite easy to obtain 6-10 minutes of clean recording. Why do you have to find it? Not sure what the use case is here...?
abdljasser2
Hello!

Immediate use case would be sampling. Say you like a certain sound in a song and would like to use it as a starting point for your own sound patch.

I also believe that transfer learning has benefits even for making great sounding instruments in cases where you have access to lots of data. That’s my intuition at least.

At the very least, it saves you a lot of memory/bandwith. Instead of having one large model per instrument you only need one large models with a few extra instrument specific weights.

willis936
This reminds me of a task in my list that has been sitting there for nearly a decade:

  instrument FIR from song (justice - let there be light)
Here is the spectrogram of the sound I'm talking about:

https://imgur.com/kmtoMkd

It's pretty easy to filter out the drums since most of the energy is in other bands. Looking at the spectrum again I don't think a simple spectral replication will nail the sound right. It looks like there is some sort of beat phenomenon that isn't present at all center frequencies.

not1ofU
I dont think I understand what you mean, but, if I do, then you could look into using spleeter. It separates musical stems.

https://news.ycombinator.com/item?id=21431071

https://github.com/deezer/spleeter/wiki/2.-Getting-started#u...

willis936
The task I gave myself was to subtract out the drum beat (the song graciously gives the isolated loop before the instrument comes in), then mix/baseband the instrument to whatever frequency I wanted. If all went well I would make a complex FIR filter that I would pass tones into.

This model assumes the timbre is independent of the tone, but I can see now that this assumption is quite wrong and something more complicated (like this ML modeling) would be needed.

squeaky-clean
That synth is extremely distorted post summing of the voices. (That whole album has so much distortion, it's lovely).

So not only is timbre not independent of frequency, summing multiple notes is also non-linear. The "beating" this causes is most obvious on the second chord to play. This beating is not consistent as the notes change, it's based on the difference in frequencies between the two notes being played.

pizza
Maybe 2nd-order spectral relationships would get you a bit closer, e.g. bispectrum / bicepstrum
[deleted]
(comment deleted)
capableweb
Where does the training start to "fade off" in terms of "time spent" and "results achieved"? It seems 1 second vs 16 seconds have a dramatic change, but what about 50 seconds vs 3600 seconds (1 hour)?
abdljasser2
I think this is a very interesting question which I currently don’t have the answer to. This is something we hope to answer in the upcoming paper.
syntheweave
As a sample-user, it would be great to have this available in the toolbox.

Just reusing the original recording of a sample is equivalent to drawing a photorealistic tracing of an image: it represents a ground truth, but it's not illustrated in any particular artistic direction. And this makes the multisample libraries available today akin to "dry references" - they can be convincing as reproductions, some of the time, but you're stitching them together like a collage of photos.

If you throw the sample into a synthesis engine you can push around the parameters, crossfade it into a loop, add some envelopes, modulation and layers, and make it a uniquely stylized instrument, and this is one way to take the source material to a new place by forgoing some realism.

Doing the synthesis through style transfer helps move it in a different direction: it gets outside the bounds of directly sequencing performance parameters and makes the performance a little more like an effect, helping to glue the sound. And I think that could be really cool if applied to arbitrary source material.

al2o3cr 

    Say you like a certain sound in a song and would
    like to use it as a starting point for your own
    sound patch.
As a bonus, you might be the Lucky Winner of a copyright suit that eventually establishes a whole new area of case law.

Yeah, it would be ridiculous and unreasonable - but so was "I copyrighted these three notes in a row so pay me naow" :(

dimal
Just want to chime in and say that I would love to have this ability. Will you be adding more documentation on how a knowledgeable user could use your library to accomplish this? The docs are kinda sparse and I'm not sure how I could actually use it.
samstave
Rather than doing a melody, what happens when you train it with a bunch of single notes of the instrument for each note.

Then give it a melody and have it play the melody based on its library of notes for a given interment?

kristianbrigman
this is closer to what Redmatica (now Logic) AutoSampler used to do.... you would just play notes (separately by space), it would split by space, detect the root note for each sample, and then make a multisample out of it.

but actually trying to reproduce a real instrument involves legato between notes, handling pitch and timbre changes as you play, etc. So a much harder problem.

samstave
Thanks for that info. Familiar with Logic, didnt know that history though.

Thanks

rhizome
>Say you like a certain sound in a song and would like to use it as a starting point for your own sound patch.

Isn't that resynthesis? I mean, this could certainly be an addition to the vocabulary of resynthesis techniques, but is there something more categorical about it?

pizza
Differentiable resynthesis. You input a desired goal, it tries to achieve it, through (essentially) trial and error. Then the artist can focus on imagining goals, rather than fiddling with knobs :)
cellover
Why in the name of God would a site prevent me from using shortcut nav to go back in nav history?!

I can not use Cmd + left on a mac, is this a Notion thing?

capableweb
I didn't understand what you meant first, clicking on my back arrow on the physical mouse, or the go back icon in the browser and Firefox would go back like expected. Then I tried Alt+Left arrow on the keyword, and it didn't work! But it wasn't just not going back, like when you have a "too many history.pushState()" bug that fucks it up for you. Instead it went to the bottom of the page. Then I noticed it was selecting some block in the bottom right and that it also happens when you just press left arrow on the keyboard without Alt.

So TLDR: bug with some selection/focus thing on the Notion page. Poor execution, I rate their implementation of "static notebook HTML pages" 7/10.

cellover
Oh right I see, I did not have the time to debug when I noticed this issue, thanks for the heads up!

Also sorry for parasiting the comments here, very interesting research! It would have been interesting to record original samples in the same settings; we can hear the flute having much less reverb than the 1st saxophone and it could be helpfup to have a common room size / reverb time to help comparison.

nyanpasu64
Does this work on synthesized sounds (like analog or physical modeling rather than samplers)? How does it handle the complexity of piano timbres (notoriously difficult to synthesize)?
abdljasser2
Great question! I think this approach (and DDSP more generally) could work well for most instruments if you make some adjustments to the synthesis architecture.

The current architecture imposes some strong assumptions on the source (such as harmonicity) which does not necessarily apply to all instruments (bells for instance). The harmonic + noise model is not that good with transients either (which is a key ingredient in piano sounds) but of course you could solve that through different means.

radarsat1
nice, i worked on something similar some years ago but this seems more sophisticated. i had a lot of difficulty with atonal sounds especially because i was trying to take a kind of "neural wavetable" approach, but overall i found the results intriguing nonetheless. curious to check out more recent efforts. i recall the DDSP paper and assumed those ideas would apply quite nicely to this problem.

[1] https://arxiv.org/abs/1806.09617

abdljasser2
Very cool! Thank you for this.

Speaking of wavetables, have you seen this relatively recent work?

https://lamtharnhantrakul.github.io/diffwts.github.io/?

radarsat1
oh nice, i have not. thanks i've got some catching up to do! i see that they used the nsynth dataset, i didn't realize it was publicly available. i recall that the nsynth paper came out just as i was finishing that work, you can imagine i felt a bit scooped ;). (but nsynth was far more impressive so what could i say..)
skykooler
Have to say I'm disappointed with the chosen excerpts in the "More examples" section - they do not show very much of the generated instrument's ability (the last one is just a single note!)
abdljasser2
Thank you for the feedback! We will synthesize longer excerpts in the future.

For the time being there is a colab where you can play with a pretrained model.

squarefoot
Any chances this could turn into a MIDI playable instrument with some pre-trained models plus the possibility to submit user generated ones?
abdljasser2
Short answer is yes! Previous work has shown that we can obtain very good results from controlling DDSP models from midi input. The solutions I am familiar with employ a two stage approach where the first stage takes midi and turns it into control signals (pitch & loudness contours etc..) and the second stage turns the controls signals into audio (like the particular model I discuss in the blog post)[1][2][3]. I actually think that the first stage could also benefit from the transfer learning techniques we discuss in the blogpost.

In terms of actually releasing a MIDI playable VST plugin I believe that Magenta have something like it in the works[4]. I hope that it will come with some ability for users to quickly create their own instruments, presumably using a transfer learning technique similar to the one we have presented.

Real-time rendering poses multiple challenges. For one, some instrument sounds occur before a note properly onsets (for example the sound of the fingers pressing the keys of a saxophone occurs before the first note of the piece). Secondly, the research models are quite heavy and considerably more compute intensive than a standard VST instrument which poses a problem if you want to use it inside a DAW. I think this latter problem can be solved with some clever engineering and the general trend of hardware being more and more accommodating to machine learning applications.

[1] https://erl-j.github.io/controlsynthesis/#/ (Our previous work) [2] https://rodrigo-castellon.github.io/midi2params/ (Focuses on realtime rendering) [3] https://arxiv.org/abs/2112.09312 (Magenta's recent paper on the subject)

mkr-hn
While interesting, I'm skeptical an algorithmic approach will ever come close to a decent wavetable-based synth (using samples/noises from the real instrument) or scripted sampler instrument (like Kontakt). It might help if you can't otherwise find an existing synth/sample-based instrument but can find it in use, but those are few and far between.
uoaei
"ever"?

I am not usually on the techno-optimist side of things but I have a hard time believing, after observing the progress of SOTA in the past decade on the big problems explored by ML research, that we are more than 5-10 years out from exceptional generative samplers based on tiny snippets of audio.

mkr-hn
I don't know what SOTA is. Are we still on-topic here? The context is realistic instruments. Synths can already produce timbres pre-electronic instrument designers could only dream of.
uoaei
"State Of The Art"
uoaei
That's true but those still need to be hand-designed. The point of the ML-based approaches is to make it easier for users to explore the space of sounds, rather than setting up a pre-defined API (knobs, switches, etc.) for modifying the sounds manually.
graphpercolator
I don't think this is even the problem though. We have amazing physical modeling synths but the problem is it is not "playable" with a keyboard in the sense of being able to control all the needed variables.

You basically need a breathe controller to make a realistic playable sax sound otherwise you can't control the variables in real time that are needed.

Personally, I want to hear synth sounds that are not even trying to be reproductions of acoustic instruments. If I want to hear a sax sound I want to hear someone who is really good on a real sax and who knows how to actually play a sax.

olau
I have so far not heard a convincing piano synthesizer, i.e. one that would fool a pianist.

Pianos are actually really easy, input-wise. You have attack speed, when you lift the keys and pedals, and that's it. But simulating the audio computations done by a physical instrument is just really difficult.

There are some physically-based (spherical cow physical) synths that sound more alive than the usual sample-based stuff, but their timbre is not right.

If I had more time, this is what I'd be working on right now. There's a market for a great piano synthesizer.

ratherlongname
How did you create the image (in 'How does this work')? Looks super clean.
Minor49er
Impressive. This is really cool. To my ear, some of the synthesized results sound a bit closer to a stringed instrument at times, but it doesn't sound out of place. I just thought that was interesting
kazinator
The use of a technique from speech synthesis must be what helps the transposed passage not sound like a dumb sampling rate change. If you sing a higher notes, "formants" (frequency response profile of the vocal tract, roughly) stay the same.
8bitsrule
Interesting study. But conventional vis-a-vis all the unexplored timbre-space out there. I'm wondering if it'll be modifiable to 'morph' between two timbres in finite steps (by mod'ing partials). So you could play on any of those steps, or float around, or gliss along them.

Flute+tuba => fluba. Fluba+violin => fluvioluba. And so on. (Harry Partch also wonders.)

ipnon
The crux of the DDPS architecture seems to be allowing a model to train the many parameters of the sound synthesizer. You would never be able to create a truly realistic piano from a software synth by merely tuning the knobs because there are too many near-imperceptible minor fluctuations in the wave form, like the way the vibrations in aggregate vibrate the wooden piano housing. Training on real life examples, in this case an impressively short clip, gives the model enough data to train these near-imperceptible variations into the synth parameters.

https://magenta.tensorflow.org/ddsp

tgv
Bit late to the action, but do I understand correctly that this trains a harmonic synthesizer? That would explain why the reverb in "How does the amount of training data affect synthesis fidelity?" sounds a bit off, as it would get synthesized on unrelated input.

But what I wanted to ask: is the final model actually understandable or visualizable, e.g. as an envelope per harmonic that depends on F0, loudness and F0 confidence?

Anyway, I'm impressed.

abdljasser2
I can't say for sure but I think the issue with why the reverb sounds off in that particular example is that the reverb present in the recording has a longer decay than the maximum reverb duration I set for the experiments (1s). I will set it longer in the future.

Yes, that kind of visualisation could be performed. The authors of the prior work we are building on (DDSP) have made some great visualisations which I think you will find useful! Here: https://storage.googleapis.com/ddsp/index.html

shannifin
Exciting stuff! I'm especially excited to see where this area will lead in regards to strings; I have yet to hear a sample string library produce a decent sounding classical symphony, at least without a lot of dynamics finessing (which, as a composer, I don't want to do; that's what human players would do naturally).