Show HN: Mofi – Content-aware fill for audio to change a song to any duration (mofi.loud.red)
I worked on a web service that allows you to import a song and define a target length that the song will be shortened or lengthened to. It does this by analyzing the song and finding repeating audio patterns. This is helpful for making any song match a video or performance with a set duration. You can also specify areas of the song to prefer or avoid. An example is available here: https://mofi.loud.red/edit/8bd3fdf780f8c3927e41029f3b957f8a7...
The cool thing is that after the song is analyzed on the server, the client can recompute and preview the results completely client-side through an implementation that uses Web Workers and WebAssembly. The audio previewing uses Tone.js. I am thinking of writing up some more details about the implementation in the future.
I'm still working on a way to explain this easily, but I like the idea of carrying over the concept of content-aware fill from images to audio.
Please let me know if you have any comments or questions!
170 comments
[ 2.2 ms ] story [ 204 ms ] threadInteresting, as the primary use case for this seems to be something along the lines of editing music to fit into video projects.
This is perfect for TikTok videos!
Will definitely be using your tool. Very cool!
> I am thinking of writing up some more details about the implementation in the future.
Please do :D
I don't remember exactly but back in the radio days, songs had to be a very specific length to be a pop single.
Where do you draw the line between acceptable "edits" vs unacceptable?
There are so many tools to customize and make your audio experience your own. Ultimately my ears are my own and I love living in a world where I can precisely control my audio experiences.
https://youtu.be/XCJs_eAkNS8
https://www.youtube.com/watch?v=ZYT9vQLxt5c
https://www.youtube.com/watch?v=n88U4n3Z3Iw
https://www.youtube.com/watch?v=YMF5djy3HNE
For people who know every beat of Strobe, it sounds odd when things don't hit the same way, but at the same time it's kind of like a unique live mix.
But for today’s music, shortening the 2010s/2020 already shorter lengths would mean a song might not be more than a minute in length. On average, full unedited tracks today end up being a bit shorter than they used to be, solely due to the economics of streaming. Rather than paying for the content second by second, it is done by paying per track play. The result is a lot of 2 minute tracks, which were produced with the “verse” parts getting jammed together into the “chorus” with no break in vocals, which also uses pitch adjustments, “the “bridge” is an afterthought that is terrible, or more recently, nonexistent……… Instrumental solo? Anyone? Bueller?
Music is no longer anticipated, budgeted for, and purchased on launch day with great fanfare. We have grown accustomed to the idea that we should have everything available at our fingertips, and as a consequence of this we get exactly what we pay for.
I think music suffers even more so because we're all so tuned into having the best at our fingertips that if a single moment in a song isn't to our liking we can skip and forget about it completely - i think this fuels the fast-short song market, easier to saturate with many short songs and get listens rather than to work/slave on a longer more intricate piece.
Back in the day, mixtapes with songs were slaved on and cherished, today slaving over something is seen as a negative quality.
I do sometimes worry about this. Some of my favorite songs growing up were ones that I didn't care for initially but which grew on me only after listening to it repeatedly. Now it's easy to dismiss anything that doesn't hook me right away. Thankfully albums are still being released, and you can force yourself to listen though them, or keep more of the "meh" songs in your playlists for longer periods of time, even adding them back into rotation after a while.
I don't think I'd give up the variety we have now though and go back to only having what songs are pushed at us through top 40 radio or the limited selection found in local stores. I get my music now from countries all over the globe. Finding old stuff all the time I'd never heard and new things just released.
We can still invest in the music we listen to and be rewarded. We just aren't forced to, so it needs to be deliberate.
Fast forward to something like Radiohead's Kid A and you have orders of magnitude more complexity going on the track. So many different sounds layered in very complicated manners. Its almost like a classical composition how there are motifs, movements, different emotions being evoked, but with any sound imaginable compared to what few are possible from the orchestra ensemble. A song from Kid A is just a part of the greater album itself. Nothing was made to stand on its own between radio advertisements.
It seems these days we are reverting to how commercial music was always made. Very commercial studio focused with the artists removed from production. Generic lyrics written by low paid songwriting staff and same old tried and true chord progressions we've heard forever. The artist is a brand and a sexy person meant to sell products versus someone particularly talented with an instrument or with songwriting skills.
It is easy to create a supercomlex composition, that’s what classical music have been doing way before 1970s. But to capture the minds and hearts of many people with simple things, now that’s the real mastery.
I'm sorry, that's the best I could do.
Identifying the good from the rest in such a place as Apple Music is a miserable experience. To Apple, they also make more revenue from their prioritization of marketing similar sounding two minute tracks in this “nouveau pop” format, backed up by a small amount of older superstar artist anniversary editions. Good original new music never makes it to the featured content sections.
And Genres and algorithms are a mess. It applies across the board to all music and is really a problem for the flavors of House, Techno, and other genres that are simply labeled “Electronic” or “Dance.” I’m getting Avalon Emerson one minute, Bicep the next, to be ruined by corporate mass marketed deadmau5, follow by a Bad Bunny remix….worse is the algorithm which thought I might like corporate edc-esque superstars and poorly autotuned remixes- though I have not added a single song to my collection in 20 years of digital music consumption…
Can we do better than “Dance” and “Electronic?” Of course they could, they haven’t. One must go elsewhere.
For House and Techno, this dearth of music discovery and search-ability methods by streaming companies makes room for independent music station alternatives, like Fault radio, or gives a reason for one to seek out artists via other means, like going to independent music festivals like Sunset Campout, Honcho campout, and other events highlighted on Resident Advisor, a poster in a nightclub, a text message listing the warehouse location.
(Apple Music did at least bring back Beats in Space.)
I think about the proliferation of streaming, and how it actually makes finding new content difficult for people who are not familiar with those other means of distribution.
- We have deprioritized the concept of the local radio station, now what was alternative rock is a rebroadcast of an AM Sports broadcast (i.e KFOG)
- the death of the sale of “singles,” made it a cheap entry point for people to experience a new artist. Releasing a track on Spotify doesn’t feel as substantial.
- and exclusively agreements contract provisions with corporate entities that engage in predatory practices that force up and coming artists to choose between performing for their fans at local venues, or extending their potential reach by putting their name on the bill for Coachella, sacrificing potential shows for a few hundred of miles away and several months on either side of the festival.
- Additionally entertainment conglomerates like AEM and LiveNation are increasingly becoming the owners, or managers of, the entertainment venues in cities around the world. Similar exclusivity agreements can have a significant negative impact on unaffiliated independent venues ability to compete.
I can’t speak for anyone in particular from the Gen Z or Alpha generation. I think for them, it’s all Apple or Spotify, music festivals if they can afford them, sharp discerning music choices on TikTok if they feel the need to branch out… and the question is - do they?
The norm is now a fully mass market formula that is almost impossible to break through… and the effect of this puts the chill on the ability to link good music with committed audiences.
I think I mostly explained how I feel there…
I'm just barely young enough to be a zoomer, so maybe my view will be interesting. From my perspective my ability to find "good" music is better than it ever would have been in the past.
I find music suggestions from forums, review sites, subreddits, friends, online people I follow, etc. Then I can immediately listen to it with no effort or expense. I found my favorite album of all time from a random comment someone left on an internet thread.
To me the idea of having to wait until a local radio station played a song, and then make a leap of faith on purchasing the album seems like such a worse experience.
Similarly, in terms of creating music, it has never been easier to learn, create and distribute your own music. The rise of the internet has made it so much easier to find a niche communities of people making incredibly diverse and experimental music together.
In my mind, streaming sites have two roles and do a very good job at both:
- make all music as accessible as possible
- suggest music for people who want to "passively" listen to music, which is how the average person has always listened to music (and that's a completely legitimate thing to want and enjoy)
I see a lot of "the death of cinema", "the death of music", "the death of video games" takes around, and I can't help but feel like these views come from people who have lost track of where the "niche" communities has moved on to and then feel despair when to them it looks like the "mainstream" is all that exists.
I adore music, always have and follow what I hope are "real artists".
I haven't bought physical media for several decades and often buy albums weeks or months after release.
I think you've sliced this wrong.
But I mostly agree with you - I get very excited for artists that I really enjoy releasing new work, and "line up" so to speak by listening to it the day it releases on Spotify.
A few days ago I was given the task of creating a corporate video - just a rolling slideshow for a shop-window display. Then suddenly it was going to go on YouTube as well - so needed some music! I found a suitable track but needed to edit it for length so its closing chord coincided with the credits card at the end of the video.
This tool might have saved me the bother of splicing the music in Audacity.
https://youtu.be/OvC-4BixxkY
and that’s just the first random 2021 song I can come up with within a second.
I hit upon the plan of taping stuff off the radio onto 1/4", and then I could splice a not-talked-over beginning onto a not-talked-over ending.
Later, I worked out that I could extend or shorten tracks, particularly if I could get a tape of the instrumental version, using the same trick.
No-one uses tape and razors these days, but it was good fun.
Tonight I was at a Melvins show in Amsterdam. It is their 40th anniversary your, amazing! I’ve also seen Einstürzende Neubauten a few months back. Back in early 90s everybody was talking how the Stones were still playing for so long. But nowadays so many bands are still going.
I don't see any of the effects you describe on my feeds (tidal; previously apple music). Perhaps you need to switch to a service with a better recommendation algorithm, or nuke your personalization profile and start over.
I'm guessing that's the timeout?
I think it was https://eternalbox.dev/ since that's all I can find on Google. But that site is down.
He created some excellent products from the Rdio API, and later Spotify … and I believe his analysis engine ended up being the foundation upon which Spotify's _play more tracks like these_ capability is based.
Looks like he's moved over to publish on Substack—there's a recent(ish) post reflecting on 10 years of Infinite Jukebox: https://musicmachinery.substack.com/p/the-infinite-jukebox-1...
Edit: I remember a multitrack file format from the past that allowed following both a composer-defined path as well as random/infinite ordering of sequences, it was called digimpro: https://www.kvraudio.com/forum/viewtopic.php?t=102403
Did I over-attribute the above to Paul? I didn't realize he wasn't a founder.
https://ismir2010.ismir.net/program/tutorials/index.html#tut...
https://musicmachinery.com/2010/08/06/finding-a-path-through...
>Tutorial 4: Finding A Path Through The Jukebox – The Playlist Tutorial The simple playlist, in its many forms – from the radio show, to the album, to the mixtape has long been a part of how people discover, listen to and share music. As the world of online music grows, the playlist is once again becoming a central tool to help listeners successfully experience music. Further, the playlist is increasingly a vehicle for recommendation and discovery of new or unknown music. More and more, commercial music services such as Pandora, Last.fm, iTunes and Spotify rely on the playlist to improve the listening experience. In this tutorial we look at the state of the art in playlisting. We present a brief history of the playlist, provide an overview of the different types of playlists and take an in-depth look at the state-of-the-art in automatic playlist generation including commercial and academic systems. We explore methods of evaluating playlists and ways that MIR techniques can be used to improve playlists. Our tutorial concludes with a discussion of what the future may hold for playlists and playlist generation/construction.
The EchoNest SDK was hosted on google code but this seems to be where it lives now:
https://github.com/echonest/remix
https://echonest.github.io/remix/
Examples:
https://github.com/echonest/remix/tree/master/examples
More Cowbell, just 113 lines of Python code:
https://github.com/echonest/remix/blob/master/examples/cowbe...
EchoNest's technology is fantastic, and it's no surprise Spotify bought them and put it to good use.
Here's an email I wrote about it when I discovered it in June 2010 (linkes updated to archive.org):
Subject: Just found and assimilated a Python music analysis library!
This is great stuff!
https://web.archive.org/web/20100623053057/http://the.echone...
Perhaps you have heard of some of the cool recent demos by EchoNest, which have been covered by boingboing/reddit/etc, that use their music analysis library (and are implemented in just a few lines of Python on top of their modules):
More Cowbell -- Adds cowbell or Christopher Walkin to music, on the beat:
https://web.ar...
There is this version now, but it does not allow custom uploads anymore: https://eternalboxmirror.xyz/jukebox_index.html
Muchos Gratias to whoever keeps that spinning. I've been itching for infinite Jepsen for a while - but discovered the original site was very dead.
https://publicinfrastructure.org/podcast/80-nick-seaver/
He just butchers an otherwise perfect song!
Will report back with results.
Edit: Results are great! https://voca.ro/13ar1g88LSKK
A less rosy scenario is like Spotify generating music to get around having to pay artists.
I like to have multiple hour continuations of songs which I use to help me fall asleep. In the past I've made my own, but I'm terrible at audio editing. It is a ton of work and, for me, really hard to get right.
After a little bit of experimenting with Mofi, it seems to do a very good job and selecting when to repeat a sections.
[edit]
Here's my first attempt for a 30 minute song:
https://mofi.loud.red/edit/ebbf4b410181aa767152945cbb6a2d679...
You can use this example though if you just want to try: https://mofi.loud.red/edit/8bd3fdf780f8c3927e41029f3b957f8a7...
"Not Imported You haven't imported this file yet. Import it first to use it."
There are a few songs I love, but some of the best parts seem to be "buried" under other instruments.
https://github.com/Anjok07/ultimatevocalremovergui
It needs to be generalized to video content, but the task is a bit easier.