Show HN: Improved freemusicdemixer – AI music demixing in the browser (freemusicdemixer.com)
Last time I showed free-music-demixer, which people seemed to enjoy. It was a static website with a Javascript + WASM module to perform music demixing (or music source separation) using an AI model UMX-L (Open-Unmix) running client-side in the browser.
Since then, I have overhauled the project and made several improvements:
- The demixing/separation quality is higher now, since I implemented the missing post-processing step
- Memory usage is lower now by performing a custom segmented inference with a streaming LSTM, which should allow larger tracks (or, dare I say, arbitrarily-large tracks)
- There is a batch upload feature now to demix an entire folder of songs (and provide zip files of the stems)
- There are now dev logs printed to the website to show the progress better
41 comments
[ 5.1 ms ] story [ 177 ms ] threadThe model I'm using is called Open-Unmix (https://github.com/sigsep/open-unmix-pytorch). In 2021, there was an update to Open-Unmix to include new weights, UMX-L, which made it perform better than it used to on the older weights (UMXHQ).
In the grand landscape of music demixing, I don't think UMX-L is near the top anymore.
_However_, the demixing performance of freemusicdemixer.com is very close to the full PyTorch performance of Open-Unmix UMX-L, despite the tricks I needed to get it working in the browser, such as splitting up the inference to operate on segments of the song, or making the LSTM operate on streaming segments rather than holding the entire track in the LSTM memory.
In my first release, I loaded and did inference on the entire track at once (like the PyTorch model), which frequently crashed or exceeded the 4GB WASM memory for medium or large-size tracks.
We have since discussed my projects a couple of times! I even got the idea for a streaming LSTM from him.
I think music demixing in general owes a lot of thanks to Open-Unmix and co (https://github.com/sigsep), who have relentlessly been publishing open-source models and related code (source separation metrics, dataset loaders, etc.) for years, and who blew the industry open with their MDX 21 [1] and SDX 23 [2] AI challenges.
[1]: https://www.aicrowd.com/challenges/music-demixing-challenge-...
[2]: https://www.aicrowd.com/challenges/sound-demixing-challenge-...
Unlike similar products, it’s free to use and doesn’t store your data. All processing is done in your browser, and your files are never uploaded anywhere. It runs well on computers and very slowly on smartphones; user beware."
I'm impressed by the fact that this was the choice made. I can only imagine it also helps keep the operational costs down, as well as liability for copyright and what not since they never become in possession of the content. However, it also means they "lose out" on a possible continuous source of training data which other less ethical evilCorp type companies would not pass up on
Basically, the _only_ special thing about freemusicdemixer is that it runs client-side, because I'm better at writing C++ for a pre-trained network than I am at training new neural networks. However, it's a cool advantage so I'll keep promoting it as the distinguisher of my "product" (since I didn't create or train the underlying neural network, I just consider it a clever WASM frontend).
The operational costs are precisely $0 since it's a static website (aside from the domain registry).
Then, when one's project idea is validated, they could then used a paid stem separation service with higher quality and commercial licensing.
That's why I added some sentences on the website to solicit partnerships for targeted advertisements, to see if any pro demixing companies were interested.
See also: https://www.stemroller.com This runs as a local app on Windows and Mac.
Thanks for sharing! I had no idea this was a thing.
>plus you can run it with GPU support as well!
Open-Unmix also originally runs on the GPU like it was intended for, since it uses PyTorch just like Demucs.
I'm curious about using WASM + WebGPU to add GPU acceleration to this project, though.
I use it for removing background music from movie clips so I can remix them and add alternate background.
e.g. I want it to show up when I search relevant terms like "free stem separation" "free music separation" "free music isolation" "music stem separation" "music source separation"
[WASM/C++ 18:07:24] Getting waveforms from istft [WASM/C++ 18:07:25] Copying waveforms
- bass, drums, vocals, other, karaoke.wav in the Single track window (at the bottom: Demixing outputs...)
- song_1.zip, song_2.zip, ... in the Batch window (at the bottom: Batch outputs...)
Like so: ``` Demixing outputs... bass.wav drums.wav other.wav vocals.wav karaoke.wav ```
There should also be a Javascript message on the left like "Preparing zip" or "Preparing stems for download"
I can make the segment size smaller (right now I'm using 60s segments).
On multiple files.
Not just free, but local, without installation, and generalized to common use-cases too? This is a standard of development that I aspire to, so thank you for being a great example!
Dev appreciation aside, I also record music with my long-distance buddy, so we often find ourselves trying to use midi recreations in order to get at least a passable version of timing and range that we can both practice from until the demo compositions have been laid down. It's often pretty far off the mark from the original track, so we will make fantastic use of this utility. Again, thanks so much!
it is probably the most useful application of the AI things I have seen AND it does as advertised on the box. nice work on the project.
Not to mention arranging or just notating songs.
Would love a VST frontend as well so I could drop a song file into a plugin and have it spit the stems out directly into my DAW
I would recommend to polish umx.cpp and put converted weights on huggingface, make it as effortless to use as whisper.cpp etc.
I could (probably) improve umx.cpp but freemusicdemixer.com is supposed to be the "effortless" compile-free version of it.