Show HN: Free AI-based music demixing in the browser (sevag.xyz)

I started down this road, and have amassed a small collection of files made up of just the first ten and last ten seconds of demixed tracks of every song in my library

I hoped to put together a system that compares the backs to the fronts and lists the output to find cool transitions, but I have no idea how to actually "grade" the similarities

Beyond basic BPM matching on the drums tracks, nothing I've tried has made for anything really compelling (sounds random... :( )

In a few years from now DJs will have buttons on their pads to extract voices, beats, melodies etc realtime. Could result into an interesting new style of club music.

Well, the naive way to do it would be to detect the onset of each beat by assuming it corresponds to kick drums on fours, and then speeding up or slowing down the track you're mixing into to correspond with it, timing it so the fours line up. This would be very easy to implement and over a longish crossfade (say a couple of bars) would be fairly convincing.

Using AI you could probably intuit the "pulse" of the music to find the first beat of the first bar of a measure, and sync tracks up so that they mix in a place that makes musical sense.

Gave it a shot - as the webpage warns, it does take a bit off time. The results are indeed impressive, although the separation isn't perfect.. each division (drum/bass/vocals) has slight echos of each other (bass line occasionally includes a slight muffled vocal, vocals include some snare, hi-hat, and strings). It's a great starting point though.

Demucs [1], one of the leading/SOTA systems, has an experimental 6-source model, `htdemucs_6s`, which adds piano and guitar:

>We are also releasing an experimental 6 sources model, that adds a guitar and piano source. Quick testing seems to show okay quality for guitar, but a lot of bleeding and artifacts for the piano source.

I also believe Audioshake [2] (a company in the space) is doing guitar separation as well.

1: https://github.com/facebookresearch/demucs 2: https://www.audioshake.ai/

The primary challenge is the availability of sufficiently large datasets with many separate stems/tracks. Those that find a way to build that, will have the ability to churn out such a model using existing architectures.

The companies doing AI mixing have a huge advantage in this area.

I have been testing and using it on Firefox 113. I wonder if it's related to the maximum memory of WASM. I compile it with `-s MAXIMUM_MEMORY=4GB`, which is supported by Chrome but maybe not Firefox.

Does the developer console say something like "Aborted" or give a memory error in Firefox? If your clip is big enough that it uses >2GB but <4GB of memory, that could explain why it works in Chrome.

Feature is called "stems" and its also available on most major dj apps like Serato and Rekordbox

anyone know if this kind of real-time filtering what is done w Apple Music’s karaoke feature?

I tried on Firefox now, it also stopped at 7.5%, my previous attempts were on Edge Chromium.

Retry. It'll work. (Brave tested)

I see that error but in my own testing it never affected the outputs. However, download weights should be working. It's not exactly an interactive progress bar, it just waits until all 45 MB are downloaded until it sets the progress bar to 100%.

Good question! So, I wasn't even thinking about WASM to begin with. When I saw llama.cpp and whisper.cpp on the front page of HN, I found the idea exciting - instead of neural networks being magic, I wanted to copy the ggml idea of parsing the PyTorch weights file myself and rewriting the inference code in a lower-level language than Python (or, it's even more accurate to say PyTorch, since there is so much matrix heavy lifting e.g. broadcasting or reshaping that is done for you automatically).

That's when I wrote umx.cpp [1] (which is what this site is based on).

On an unrelated project, a friend of mine mentioned WASM, and as I looked into it a bit more I thought trying to compile umx.cpp to WASM would be a great idea, since I only use Eigen (which is a header-only library that only depends on std).

1: https://github.com/sevagh/umx.cpp

What about Melodyne?

In the free-music-demixer repo, there is a `file_demixer` utility [1] which you can compile using the CMakeLists file - it will use the same quantized weights and produce the same "lower quality" output as the website. Clone the git repo with submodules because it compiles the vendored libnyquist library to load audio files.

Of course, you can always run the upstream umxl model easily:

``` pip install openunmix # pulls in pytorch ```

Installing openunmix installs the `umx` cli:

``` $ umx --help usage: umx [-h] [--model MODEL] [--targets TARGETS [TARGETS ...]] [--outdir OUTDIR] [--ext EXT] [--start START] [--duration DURATION] [--no-cuda] [--audio-backend AUDIO_BACKEND] [--niter NITER] [--wiener-win-len WIENER_WIN_LEN] [--residual RESIDUAL] [--aggregate AGGREGATE] [--filterbank FILTERBANK] [--verbose] input [input ...]

UMX Inference

positional arguments: input List of paths to wav/flac files. ... ```

It will download the weights automatically for you and demix at a higher quality than my site, for two reasons: * Unquantized weights (small impact) * Post-processing step (bigger impact)

I tried to tackle the post-processing step in my C++ code (which would win ~1 dB in quality across all targets) but it's too tricky for now [2]. Maybe some other day.

1: https://github.com/sevagh/free-music-demixer/blob/main/examp...

2: https://github.com/sigsep/open-unmix-pytorch/blob/master/ope...

I've thought about this problem before (but I have so many projects going on that I axed it).

Hypothetically, this is how I would approach it.

I would start by forking Open-Unmix or another 4-stem model (Demucs, MDXNet). The code is oriented towards the 4 sources (vocals/drums/bass/other), which is dominant because the major datasets for training use these stems.

The Open-Unmix training code goes like: ``` x, y1, y2, y3, y4 = load_training_data()

y1_est, y2_est, y3_est, y4_est = unmix(x)

loss = loss([y1, y1_est, y2, y2_est, y3, y3_est, y4, y4_est]) ```

In your case it may be simpler like `noisy mix = clean mix + background noise`.

In that way, I don't even think you're constrained to using a dataset that has 4 stems available (which is a rare quality in a dataset).

Instead, you need a way to acquire or generate screaming and other concert noises.

Then, the new training code could look like: ``` x = load_training_data()

n_samples = x.shape[-1] # get length of music waveform

noise = generate_screams(n_samples)

x_noisy = x + noise

x_est = unmix(x_noisy)

loss = loss(x, x_est) ```

Well, anyway. That's my naive first idea of how I would approach it. But, this relies on having clean background noise/screams _without any music in it_. And I'm not aware of such datasets.

You can always run higher quality models than Spleeter, for example Demucs or the real Open-Unmix UMX-L, in their full PyTorch glory, as long as you have a computer that supports it.

I'm somewhat limited in what can run in a browser by the nature of the project.

wait, deezer? I wonder why they made something like this

+1 I had the exact same question... would love to know if there are any open source models that match the accuracy of Moises.ai