Launch HN: Soundry AI (YC W24) – Music sample generator for music creators (soundry.ai)
We (Mark and Justin) started writing music together a few years ago but felt limited in our ability to create anything that we were proud of. Modern music production is highly technical and requires knowledge of sound design, tracking, arrangement, mixing, mastering, and digital signal processing. Even with our technical backgrounds (in AI and cloud computing respectively), we struggled to learn what we needed to know.
The emergence of latent diffusion models was a turning point for us just like many others in tech. All of a sudden it was possible to leverage AI to create beautiful art. After meeting our cofounder Diandre (half of the DJ duo Bandlez and expert music producer), we formed a team to apply generative AI to music production.
We began by focusing on generating music samples rather than full songs. Focusing on samples gave us several advantages, but the biggest one was the ability to build and train our custom models very quickly due to the small required length of the generated audio (typically 2-10 seconds). Conveniently, our early text-to-sample model also fit well within many existing music producers’ workflows which often involve heavy use of music samples.
We ran into several challenges when creating our text-to-sound model. The first was that we began by training our latent transformer (similar to Open AI’s Sora) using off-the-shelf audio autoencoders (like Meta’s Encodec) and text embedders (like Google’s T5). The domain gap between the data used to train these off-the-shelf models and sample data was much greater than we expected, which caused us to incorrectly attribute blame for issues in the three model components (latent transformer, autoencoder, and embedder) during development. To see how musicians can use our text-to-sound generator to write music, you can see our text-to-sound demo below:
https://www.youtube.com/watch?v=MT3k4VV5yrs&ab_channel=Sound...
The second issue we experienced was more on the product design side. When we spoke with our users in-depth we learned that novice music producers had no idea what to type into the prompt box, and expert music producers felt that our model’s output wasn’t always what they had in mind when they typed in their prompt. It turns out that text is much better at specifying the contents of visual art than music. This particular issue is what led us to our new product: the Infinite Sample Pack.
The Infinite Sample Pack does something rather unconventional: prompting with audio rather than text. Rather than requiring you to type out a prompt and specify many parameters, all you need to do is click a button to receive new samples. Each time you select a sound, our system embeds “prompt samples” as input to our model which then creates infinite variations. By limiting the number of possible outputs we’re able to hide inference latency by pre-computing lots of samples ahead of time. This new approach has seen much wider adoption and so this month we’ll be opening the system up so that everyone can create Infinite Sample Packs of their very own! To compare the workflow of the two products, you can check out our new demo using the Infinite Sample Pack:
https://www.youtube.com/watch?v=BqYhGipZCDY&ab_channel=Sound...
Overall, our founding principle is to start by asking the question: "what do musicians actually want?" Meta's open sourcing of MusicGen has resulted in many interchangeable text-to-music products, but ours is embraced by musicians. By constantly having an open dialog with our users we’ve been ab...
102 comments
[ 2.9 ms ] story [ 66.5 ms ] threadWith respect to growth, our goal is to continue lowering the barrier to entry for people to write amazing music that they are proud of. Our initial product is focused on music samples, but by increasing the level of abstraction to remixes and then full stem generation we can literally enable more people to consider themselves to be "musicians." Adding music distribution to our platform will greatly improve virality and organic growth, since most people who make music want to share what they've made with their friends. We are confident that we can achieve our lofty goals by building a vertically integrated music creation and distribution platform.
You'd imagine you could estimate most music with this with less compute and higher determinism and introspection. Is it because there isn't enough training data for the above?
If you're interested in really exciting work on applying AI to creating synthesizer patches, I recommend you check out synplant2: https://soniccharge.com/synplant2. Their tool can load in any audio and then create a synth patch which sounds nearly identical to the input audio.
I started music DRL (https://github.com/chaosprint/RaveForce) a few years ago. At that time, SOTA was still the "traditional" method of GANSynth.
Later, I mainly turned to Glicol (https://glicol.org) and tried to combine it with RaveForce.
There are many kinds of music generation nowadays, such as Suno AI, but I think the biggest pain point is the lack of controllability. I mean, after generation, if you can't fine-tune the parameters, it's going to be really painful. As for pro, most of the generated results are still unusable. This is why I wanted to try DRL in the first place. Also worth checking
https://forum.ircam.fr/projects/detail/rave-vst/
If this is your direction, I'm wondering if you have compared the methods of generating midi? After all, the generated midi and parameters can be adjusted quickly, it is also in the form of a loop, and it can be lossless.
In addition, I saw that the demo on your official website was edited at 0:41, so how long does it take to generate the loop? Is this best quality or average quality?
Anyway, I hope you succeed.
I 100% agree with your point about Suno AI. If you're an amateur you want to be able to have the ability to control and change the output, otherwise how can you call the music your own? If you're a professional, without the ability to control you can never achieve your specific goals! This is why we feel confident in our musician-first approach.
WRT Midi generation we are absolutely considering it, but we don't think we can really offer anything unique there. We believe our ability to create natural sounding instruments is key to enabling the creation of all genres of music. With that said though, the ability to generate MIDI is #1 on our Canny board so maybe that should be next :)
Our text-to-sound model takes roughly 10 seconds to generate, and our Infinite Sample Packs are instant since we pre-compute output to hide latency.
Thank you for your thoughtful questions!
https://en.wikipedia.org/wiki/MUSIC-N
Later SuperCollider and TidalCycles led the way in live coding. For me, I just wanted a tool that could write code, compose music or design sounds directly in the browser, play with friends and have sample-level control. From the perspective of sample level control, it seems to be two extremes compared with the black box of AI.
To the thread OP, with all due respect, I'm not sure if your team is solving the right problem. You mention "We (Mark and Justin) started writing music together a few years ago but felt limited in our ability to create anything that we were proud of." but how does that indicate a problem with the tools rather than your mastery of the composition process?
Let me say it another way. The cambrian explosion I have seen in the space of bedroom pop paints a different picture, which is that a sufficiently motivated teenager can jump from wanting to write music to producing polished hits in months of focused effort, and from there, a signed record label contract. This cycle has been progressively shortening over the past decade with the improvement of at home DAW software/plugins and median quality of computer horsepower and entry-level audio hardware.
This is also not necessarily hidden knowledge -- across the many forums of bedroom producers, almost every one of them have had a phase where they believed the problem was their tools, which distracts them from improving their composition fundamentals, and which is almost always resolved by forcing themselves to write better songs with even more primitive tools. While this discovery took a bit more time when tools were more primitive, it is a process that hungry early-stage composers hit a lot earlier today given the power of tools and the expected level of sophistication they are all expected to have by the market. Indeed, I experienced this myself formally during university when my college music composition composer forced our classes to write songs with constrained pitch class sets and instruments. The constraints actually forced us to figure out how to use more primitive tools to their full potential by making up for it with creativity, rather than using more advanced tools to less potential by virtue of less creativity.
Combined with the rise of streaming audio platforms, it is also the case that the median level of conceptual polish as well as the bar for releasing a track that breaks through the noise has also risen. If every teenager excited about music can go from 0 to professional outputs in several months, then one would expect to see the results of that in the market -- which I have certainly seen.
My concern with your platform thesis is that it is optimizing a part of the composer's journey that you felt but which was not a material reality of your target power user's day-to-day for any period of time except the very beginning, and that you didn't do enough research into the market before building around a problem to solve. The comment I am replying to goes quite a bit closer towards solving what I see as the real problem: helping power users dial in closer to the sound they are looking for and already know how to get to, but simply in less time. Given that you have a professional DJ and producer on your team, I find it curious that this wasn't immediately caught and corrected for in your original market thesis.
If you do not correct this grave mistake in your thesis about the market, I believe your product is doomed to fail. You'll achieve success solving the problems of users who are doomed to never be successful musicians, but you'll fail to solve the problems of users who are on the arc towards becoming very successful, merely because you haven't made the deliberate choice to focus on them and their specific needs. If you did, I think you would arrive at a very different product.
I'm very interested in this ( although I'd rather have an API), but it's major red flag if I don't know the price
0: https://app.soundry.ai/forge
I put together my initial thoughts here: https://www.youtube.com/watch?v=nAZAWBw7c7o
Also, I love your suggestion for humming and then using that audio as input to the AI. Our text-to-sound product (called The Forge in-app) does actually support you uploading your own audio and then you can add text and other prompting to modify it! I wouldn't say that particular functionality is fully developed yet, but your point is super valid since many musicians communicate rhythms and melodies to each other by imitating instruments with their voice.
how did you determine market size? TAM etc?
You have a typo - presumably not VTS3 but VST3 :-)
This game is about product, not models.
The alpha and margin on models will trend to zero. Razor thin products like ElevenLabs will become commodity, whereas rich UI/UX will stay ahead.
OpenAI themselves will be stretched too thin to build for this market too.
There are so many models out there for all sorts of use cases. A lot of the weights are in the open and you can fine tune on top of them.
There will be no single "foundation model" that wins. There will be dozens of competing foundational models that are all racing to the bottom. The product build around them and the user base / community are what matter.
Honestly, the AI start ups YC is funding appear uniformly bad ideas. Weird.
You can't think of ANY creative uses for this? After 20 years? It's just another tool, just like early hip hop sampling -> drum machines -> midi controllers -> etc etc
Using human language as an interface to the world of music defeats the entire point of the exercise. Neither of the founders is a professional musician, it’s missing the point of what it means to make music on so many levels.
Also, it will be a real disappointment to Diandre when I tell him that his years of touring and releasing 100+ songs don't amount to him being a professional musician.
Apologies for the tone.
I'm particularly excited by the idea of gen ai creating entirely new sounds, sort of becoming its own kind of instrument instead of generating or emulating samples previously created / trained on.
Somewhat analogous to how the MPC etc. enabled a generation of musicians to chop and pitch and arrange soul samples into new types of hip hop music. Not super familiar with the history but I don't believe they thought it would be used like that.
I'd imagine a gen AI musical instrument just needs a lot more "knobs" to tweak and eventually someone will find that a particular "hallucination" sound to be interesting. Exciting times!
For me the exploration and creation of melodies and rhythms is the fun part and this takes it away from me a bit. I am not opposed to using AI in my music - I've been using Eleven Labs to create vocal samples that I've incorporated into some tracks.
I would rather have it generate a drum kit and patterns rather than samples of complete loops.
Congrats on the launch! Great to see a genAI product that generates ingredients for creative use rather than just a finished product.
What I'd be looking for is not generation of complete samples, but a tool that could coach me and assist me to achieve the sounds I am looking for with minimal technical know-how. For example, I might like to create a track with similar "instruments" to an existing track that I like. I could upload an mp3, the tool would identify the main "instruments", configure some kind of plugin for each of them, and then I could start making my own tracks using my keyboard and / or piano-roll. Eventually, I would graduate to making new "instruments", perhaps by generating variants of the ones that I have "favorited" in the past. Over time, I would become more adept at using the tools and interface available for tweaking the sounds.
Works well for me, but I have to say that I have a taste for minimalism, so the my starting point is already quite basic.
What I really want is to be able to feed a sample to the AI and have it spit out SYSEX that will reproduce that sample on my room full of synthesizers, so that I can go from Sample->Synthesizable Sound, i.e. turn the static dead of a sample into a dynamic reproduction with hundreds of modifiable parameters.
I'm guessing someone is working on this. Aphex Twin had a cute project along these lines, but I dunno where it went ..
Something like that would indeed be very cool. Hope I understood you correctly.
Like I said, Aphex Twin had a project like this a few years back:
https://fo.am/activities/midimutant/
.. but it ran into some limitations - I'd love to see this updated with modern AI techniques.
I guess we're pretty close to that .. keep an ear out!
But I think while generated samples will be somewhat unique, overall sound would be mostly generic.
A tool that generates random MIDI melodies is also "infinite", but it's infinitely crappy.
There's an inherent problem with "unique" samples, that they can't be reviewed and ranked by users, since each user only sees their own.
There's a chance a tool like this make users waste time instead of saving time, because you would need to listen to so many "generations" before finding something you like.