"On a V100, it takes about 3 hrs to fully sample 20 seconds of music."
That might make building off this project out of reach of the average engineer (you certainly cannot build that into a Colab notebook), although that necessary amount of compute is not surprising.
I would guess that on average, it takes a professional more than 36 hours ((4×60÷20)×3) to make a 4-minute audio track with original music based on given lyrics.
I don’t really see the point of this comparison. Composing, arranging, and producing a song is not a benchmark you can profile against; musicians are not performing some kind of music compute that produces a set number of music units per hour.
Speaking from my own experience, I’ve had tracks that took months to complete, and I’ve had tracks that I got to probably 90% completion in under an hour. I would propose that there’s no meaningful definition of “superhuman” for creative efforts.
Agreed. Although "professional" pop production does tend to be somewhat involved, it doesn't have to be, and total time spent could vary so radically as to have essentially no correlation to anything else.
Eh. It's built on Transformers, and people have already demonstrated considerable model distillation/compression on those just like every other kind of NN, and as they note, once you've trained a teacher model, you can probably train a wide flat model for similar results. (As I recall, WaveNet used to be similarly slow, but even without the parallel WaveNet retraining, with proper caching of repeated states, you could make it orders of magnitude faster and approach realtime.)
They added a link to a Colab notebook. The upsampling takes most of that time, so if you're wiling to deal with a noisy and compressed sounding piece, it's actually very doable.
Definitely a market for this. There are so many events that like to use music as background noise but due to licensing restrictions in music you have to be careful what you use.
The future will be AI lawyers battling for rights of AI generated music in the style of deceased artists on behalf of AI media corporations at the expense of robotic listeners.
Before all of this, we'll probably see improvised bands of deceased artists playing together AI generated music in their own style, not to mention long dead actors appearing in new movies etc. AI technology is going to give law firms a lot of work in the future.
I'm thinking of characters like Captain Jack Sparrow (partially inspired by Keith Richards) or Zuse from Tron: Legacy (partially inspired by David Bowie).
When it's less "inspired by" and more literally "0.5 David Bowie", I also imagine a lot of law firms writing letters.
One of the many many examples of why the 2006 Idiocracy docu^h^h^hmovie would well deserve a sequel. Probably even two, considering how much material we produced since then.
With http://songsmith.ms/ from Microsoft Research you just sing whatever and it tries to fit cheesy, casio-keyboard music to your mumbles to make it sound like you planned it. Of course, the real fun is taking vocals from popular songs and making casio-keyboard covers https://www.youtube.com/watch?v=wTN-ixHQ2hM
> I predict that in very near future you just write funny lyrics, select the style and vocalist you want
You probably don't even need to write the lyrics yourself, but just select any topic you want, genre and mood then entire song generated, for example lyrics generated using Artificial Intelligence
I think you're way off base. I feel like the remaining gap, in comparison to the progress it represents, is more like dotting i's and crossing t's at this point.
I mean I listened to the metal track, and I usually like metal, and I couldn't stand it. The guitar was just ... wrong. The lyrics were unintelligible, even though I had them right in front of me.
The pop song in the Katy Perry style was sort of intelligible but quite repetitive (moreso than most pop songs).
The other songs had similar issues.
I agree that it's quite an achievement, but it clearly suffers from the uncanny valley.
Weird. It loads for me and I do see visitors coming so it might be just you? Send me an email (my-hn-username@gmail) and I'll notify you when I launch.
I'm curious what this might be. I definitely like the sound of an "IDE for music composition," but your landing page gives me almost no idea at all of what it is. Screenshots or video? Or at least some description of what makes this different than all the other iOS apps that advertise "makes music easy! No experience necessary!" To me those are the biggest red flags that something is useless for actual musicians.
As a side note, I take huge issue with "Music is fundamentally unsolvable by AI". That's a ridiculous stance that sounds way too much like "humans have these soul things that are made out of magic and computers can't ever have them."
My only question with OpenAI is whether they will forever-more take existing AI research, throw $100,000s of dollars at it in training, then take credit for inventing intelligence
We release our model weights and code here https://github.com/openai/jukebox/, so you can directly build on top of them and don’t have to train from scratch
So, the music that I know the most about is dance music and all of your examples from that genre seems to have completely missed the four to the floor beat that characterizes those artists — any theory as to why that is? You’d think that the loop based repetitive nature of edm would make it simple for an ai to mimic.
I think work like this will really bring a whole new life to a lot of video game music. Today, we see some really great composers making cinematic-level music for video games, which is great. What worlds often miss is ambient sounds, a radio as you're driving or something that reacts to how you act (Actions per minutes go up, maybe the tempo does too?) without having to compose a TON of music.
I feel like the AI is rickrolling us, as it never quite gets to the chorus. It ends the first verse/pre-chorus at 0:30, goes into an instrumental, then repeats the pre-chorus, then babbles unintelligible until song end.
Can anybody explain why the researchers are attempting to generate the whole song as a single waveform, as opposed to wiring generated MIDI into some instruments and separately a singing algorithm (perhaps a bit easier than the whole bulk work)?
It's very hard to express all the nuances of real music and tonality in MIDI -- so generating raw audio side-steps all the limitations of a MIDI intermediary, and IMO, the results are absolutely phenomenal!
(BTW, there are lots of AI music generators that generate MIDI, so it's less interesting either way.)
We did work last year on MIDI alone - https://openai.com/blog/musenet/ and some early work now on conditioning the raw audio based on MIDI (early results at the bottom of the Jukebox blog). Agreed though there should be interesting results from modeling different blends of MIDI, stem, and raw audio data. Raw audio alone gives us the most flexibility in terms of the kinds of sounds we can create, but it's also the most challenging to get good long term structure. Still lots more work to be done!
Personally, I think the example "songs" are all awful. None of them would succeed on any criteria, despite the admittedly low bar for music composition and vocal performance that passes today.
This project only serves to demonstrate that computers cannot make art; only people.
> Co-written by a language model and OpenAI researchers
Researchers co-wrote all the lyrics. This is one place where reading the fine print matters. Super impressive stuff, but I also wonder what had to be tweaked.
Just know that: much of the stuff OpenAI and other research orgs put out (including mine) are heavily cherry-picked. Most of the time its pumps out gibberish, but in the off chance it doesn't it gets used as marketing material.
There is a huge chasm between more monkeys than atoms in the universe typing scripts unseen and a bunch of GPUs generating a few hundred samples from which researchers can cherry-pick.
All you have to do is click through to see all the samples and it becomes clear how incredibly cherry-picked the ones on the front page are. It is a cool project but it is very clear how much work this technology will need before it is useful in any application.
Cherry-picking is exactly what artists do best. They will want this technology as a new tool in their toolbox. I expect some future genre of music using it's successor (like autotune).
So a lot of this sounds muffled and compressed... I wonder if something like the equivalent of a super-resolution or denoising autoencoder for music would work here as a post-processing step.
Like, just pass through the network w/o style transfer, use the input and output as a training dataset.
Yeah, the test would be something that generated MIDI which gave pleasing results when connected to a good library. This reminds me of the way early DeepDream pictures all looked like a litter of puppies on acid.
In my view, attempts like this misunderstand much of the point of music. That is, to communicate aspects of human life that are deeply interwoven with facts and experiences outside of the music itself.
I don't see how any of that will be possible before we have some kind of general AI, and in the meantime I think these attempts will continue to be semantically empty, even unsettling in their emptiness.
Agreed, The "data" being used to generate real human music is the human condition ... so anything trained on a featurized low dimensional representation of that will ultimately be derivative.
Image and sound are ultimately related to feeling ... and it is those feelings that give us humanity -- not the ability to think and manipulate symbols (though that was not apparent to me until this current AI revolution)
I think we will soon find that what really makes us "human" is the shared chemo-biological composition of our selves with the rest of the ecosystem and evolutionary hierarchy.
When you cuddle a puppy you can actually smell the infant hormones on them... why... because we share evolutionary biology. How do you teach a computer to do that?
You can think of the body as "data" and the brain as nothing more than a database that let's you query it. AI might give a better database... but until it gets the same chemo-informatic data... well good luck.
Good point about the necessity of embodiment. I would go one step further and consider the environment, which is the source of evolution and knowledge, a simple environment like Atari games can't even begin to compare with the human society and world we experience. What makes us human has a lot to do with the dynamics of interacting with the other humans, an AI would need to be part of society to experience that.
> In my view, attempts like this misunderstand much of the point of music. That is, to communicate aspects of human life that are deeply interwoven with facts and experiences outside of the music itself.
I actually think you've missed the point. These attempts do not aspire to communicate aspects of human life at all. They're simply scientific and engineering endeavors that seek to answer less profound questions like: "Can computers generate music?" (Yes) and "Can computers generate music that is enjoyable to listen to?" (Not yet)
To go one step further: There are glaring and obvious technical faults in many of the generated samples (this isn't a criticism, they're better than past work!). I suspect that if you are feeling unsettled by these songs it's because of those flaws and not because they are "semantically empty".
> These attempts do not aspire to communicate aspects of human life at all.
Of course not. They, just like enough humans do already, imitate the results of "having an adventure of the soul".
> "Can computers generate music?" (Yes) and "Can computers generate music that is enjoyable to listen to?" (Not yet)
And we're talking about the question "should they?", which science can't even attempt to answer. "Play from your heart", and all that; not even best-selling artists pumping out mediocrity are above that criticism, even when they do it according to the best of their ability and conscience, and even when it makes people "happy".
Most people don't really care about deeply interwoven aspects of the human condition or semantic meaning when they listen to music, because most popular music is shallow and derivative as it is, a catchy beat and a hook and little else. When you think about the possibilities for automating creative output, you have to consider that the lowest common denominator brings the most potential profit.
I think you're missing that even shallow popular music is fundamentally about the interplay between familiarity and unfamiliarity in a way that's informed by the broader world. Sometimes it's about knowing who the performer is and how a song fits into their life and persona, or maybe it's about the way the melody and style conform to or defy current idioms, but it's definitely not about anything simple enough to be replicated in an unguided way by an AI. Like an AI could spit out a perfect 2000-era Britney Spears song tomorrow and it wouldn't be a hit regardless of its technical merits, because that's not what anyone is actually looking for in 2020.
And this is music with a different, albeit equally valid "point": to see ourselves reflected, abstracted, and find what we can still recognize. It's like a Rorschach test, or a piece of highly abstract art. Who are we to say what was going on in the mind of the artist? So often, we are absolutely wrong about their state of mind, their intention, those experiences and beliefs.
Alternatively, here, we are still witnessing art. The artist, as ever, is human: the scientists who pieced together these techniques. Theirs is the voice, if only humans can have a voice, that we hear in the work.
They are not semantically empty: they are absorbed, semantically, in the domain of the computer scientist who, through no fault of their own, could never sing before now.
> That is, to communicate aspects of human life that are deeply interwoven with facts and experiences outside of the music itself.
What does that even mean? As a counter-point I listen to heavy bass music with zero lyrics. The production value is the most important thing for me and I would 100% listen to AI generated music.
Who gets to decide what is the "point" of music? Music is a twenty billion dollar industry. An AI system that can spit out highly "realistic" and "pleasing" music can change the music industry as we know it.
I think people in the comments are completely missing the point of this work. As I understand it, and take this with a large grain of salt because I haven't read the paper, the idea of Jukebox is to take a certain style of music by a certain musician and have the algorithm sing, karaoke-style, the lyrics that are listed in the examples to the tune of that music. Think of it as a really jazzy version of Google text-to-speech. The lyrics are not written by this algorithm, it's just singing in the style of Sinatra or Lady Gaga some words that have been prewritten. It's fun to listen to and really amazing to watch it read the lyrics and decide where to put emphasis, and where not to - dragging out certain words and letting others be mumbled. Comparing this to something like IBM's rendition of a "Bicycle built for two" showcases how utterly mind-blowing this work is!
Finally, can we stop treating ever single piece of work by neural networks as a "failure" because it isn't GAI? Just because it doesn't "say something about the human experience", doesn't make it bad engineering. It's hilarious how as soon as there's some new AI work done everyone starts wailing, "where's the humanity!"
> It's hilarious how as soon as there's some new AI work done everyone starts wailing, "where's the humanity!"
Lay-people think AI refers to ALife.
Most of the talking heads would be immediately satisfied—giving none of these complaints—if they were shown an "AI" program that responds to stimuli by entering emotional states, and which learns to associate stimuli with the emotional states it has been in in the past, such that those stimuli will then become triggers for those states, and for memories associated with those states.
Such an agent wouldn't even need to use ML techniques, necessarily. It'd just need to be a high-concept tamagotchi that can respond to operant conditioning. That would already be an advance over the state of the art.
But, AFAIK, nobody's really working on ALife in the sense of "making an individual agent with a complex-enough internal model that it can statefully respond to you the way a pet does." ALife is only really studied at the very low level (C. Elegans connectome simulation) or the very high level (sociological/economic simulations using simple goal-driven agents); nobody's really working in the space "in between." (Except for the people trying to make chat bots seem friendlier, but they're mostly trying to fake it, rather than creating actual persistence-of-memory.)
I wonder why nobody's interested in medium-scale ALife research these days? It used to be a hot topic, back when it was conflated with robotics under the banner of "embodied cognition."
So basically, most talking heads would be better off playing The Sims. They'll have agents there that enter emotional states in response to stimuli. Even though it's just a fuzzy state machine.
Now, is A[rtificial] Life the correct term to use here? I feel it isn't - I'd expect ALife to be more concerned with implementing simulacra of bacteria or worms in silico, not with reasoning or emotions.
ALife is fundamentally concerned with the research on the kind of control systems that govern how organic life responds to stimuli, how those systems plan in order to maintain long-term homeostasis, how they select goals, how they allocate attention, etc.
One might say that ALife is to an event loop as AI is to a one-time query-response. AI can evaluate, but you need an ALife system in order to "think" in a continuous way.
There's really no sense in which an ALife researcher cares about recreating a full-fidelity model of biology in silico; the point is to specifically study the thinking and decision process of real agents, and figure out how to model those, in a way that the model makes the same series of decisions the real agent does in the same situations (and, therefore, must also be keeping and updating analogous internal state to the kind the real agent keeps.)
Some of those models are attempts to recreate real brains/nervous systems, but these models aren't fundamentally biological. A "low level" connectome simulation doesn't contain any model of cellular inflammatory response, cellular waste and its clearance, etc. It's basically just a brain-as-actor-model with neurons as stateful processes and electrochemical signals as messages.
An ALife researcher cares about as much about biology below the level of intracellular pharmacodynamics (sodium channels et al), as a race-car-chassis engineer cares about physics below the level of fluid dynamics. They don't need to go any lower, because they've found an encapsulating abstraction that makes all the predictions they're interested in making, without needing any lower-level information.
You misunderstand critically that this is not "singing along", it's generating the music and voice. Conditioning on lyrics is optional, and done "unaligned", eg by arbitrarily encoding the lyrics and passing them as additional input.
Indeed, the extent of generation is obvious in the ‘continuation’ mode on any track that is rather familiar for the listener (ahem Rick Astley). Besides, in the full sample browser there are tracks without lyrics.
At the risk of sounding crazy, I think this is a pretty big milestone towards some semblance of AGI (or at the very least ASI that writes songs). The fact that neural networks are even capable of producing such outputs (even when cherry-picked) is surprising. In a sense, showing off the promise behind this kind of technology provides a cohesive vision of what this could be in its final form, which in turn inspires people to work on it and fix the pending issues.
Just think about how GANs were viewed when they were first published. The common sentiment was that it was as an interesting "research contribution" that could never live up to the hype. However, the promise behind it inspired people to continue to work on it and now we're able to produce realistic human faces that humans can't tell are fake.
Kind of disappointed with the lack of classical - no Bach? I feel like it'd be easier to achieve more successful results with classical anyways, given that it's vocal-less and more rhythmic/predictable, with slower tempo.
I can imagine in a future iteration of this, writing a song, recording it with your phone, and then letting this turn it into something that sounds like a high quality production performed by a famous voice.
IANAL, but it strikes me as pretty obvious that the owner of the NN is the owner of the copyright on any works created by the NN with the important qualification that training the NN on works copyrighted by others could possibly be considered by the courts to be infringement.
Sure, but imagine a scenario when I use transfer learning. I download a pretrained model from OpenAI, make some tweaks, maybe train it on my own music, and have a michael jackson level triller hit?
In the United States, there was case that got a decent amount of publicity where the opinion was that training a model on copyrighted works is "highly transformative" and is therefore permitted under fair use.
133 comments
[ 3.8 ms ] story [ 196 ms ] thread"On a V100, it takes about 3 hrs to fully sample 20 seconds of music."
That might make building off this project out of reach of the average engineer (you certainly cannot build that into a Colab notebook), although that necessary amount of compute is not surprising.
I would guess that on average, it takes a professional more than 36 hours ((4×60÷20)×3) to make a 4-minute audio track with original music based on given lyrics.
It’s impressive that now, they “only” need to improve the quality for it to outcompete professional musicians on commercial delivery.
Speaking from my own experience, I’ve had tracks that took months to complete, and I’ve had tracks that I got to probably 90% completion in under an hour. I would propose that there’s no meaningful definition of “superhuman” for creative efforts.
Then we hear it in
- private events like weddings.
- social media creators make their own music to go with their funny videos. Cheap theme music for streamers and podcasters.
- Advertising. Shopping centres make lyrics that advertise products and play them to you as pop songs. Some bubs make their own songs.
Before all of this, we'll probably see improvised bands of deceased artists playing together AI generated music in their own style, not to mention long dead actors appearing in new movies etc. AI technology is going to give law firms a lot of work in the future.
When it's less "inspired by" and more literally "0.5 David Bowie", I also imagine a lot of law firms writing letters.
"If you want a vision of the future, imagine a human face booting on a stamp forever."
(From the last story at https://slatestarcodex.com/2016/10/17/the-moral-of-the-story...)
It would put record companies in an interesting position.
You probably don't even need to write the lyrics yourself, but just select any topic you want, genre and mood then entire song generated, for example lyrics generated using Artificial Intelligence
E.g. https://TheseLyricsDoNotExist.com
This project is very interesting, but it goes to show just how far we still have to come before AI is replacing creativity.
The pop song in the Katy Perry style was sort of intelligible but quite repetitive (moreso than most pop songs).
The other songs had similar issues.
I agree that it's quite an achievement, but it clearly suffers from the uncanny valley.
But I also know that in AI, it's that last little bit that's always the hardest.
http://ngrid.io
Launching soon.
Music is fundamentally unsolvable by AI. We'll have AI writing code before we'll have AI writing meaningful music.
Sounds interesting, would love to take a look at what you're building.
Edit: seems to be fixed now?
As a side note, I take huge issue with "Music is fundamentally unsolvable by AI". That's a ridiculous stance that sounds way too much like "humans have these soul things that are made out of magic and computers can't ever have them."
If they used on-demand AWS instances, it would cost about 1,342,623 USD to train the top-level prior. So much for reproducing this work.
We're getting closer. Music is proving to be a tough use case for generative ML.
(BTW, there are lots of AI music generators that generate MIDI, so it's less interesting either way.)
https://magenta.tensorflow.org/ddsp
This project only serves to demonstrate that computers cannot make art; only people.
> From dust we came with humble start; > From dirt to lipid to cell to heart.
That's not just a passable lyric. I think it's downright _good_.
> Co-written by a language model and OpenAI researchers
Researchers co-wrote all the lyrics. This is one place where reading the fine print matters. Super impressive stuff, but I also wonder what had to be tweaked.
Like, just pass through the network w/o style transfer, use the input and output as a training dataset.
I don't see how any of that will be possible before we have some kind of general AI, and in the meantime I think these attempts will continue to be semantically empty, even unsettling in their emptiness.
Image and sound are ultimately related to feeling ... and it is those feelings that give us humanity -- not the ability to think and manipulate symbols (though that was not apparent to me until this current AI revolution)
When you cuddle a puppy you can actually smell the infant hormones on them... why... because we share evolutionary biology. How do you teach a computer to do that?
You can think of the body as "data" and the brain as nothing more than a database that let's you query it. AI might give a better database... but until it gets the same chemo-informatic data... well good luck.
I actually think you've missed the point. These attempts do not aspire to communicate aspects of human life at all. They're simply scientific and engineering endeavors that seek to answer less profound questions like: "Can computers generate music?" (Yes) and "Can computers generate music that is enjoyable to listen to?" (Not yet)
To go one step further: There are glaring and obvious technical faults in many of the generated samples (this isn't a criticism, they're better than past work!). I suspect that if you are feeling unsettled by these songs it's because of those flaws and not because they are "semantically empty".
Of course not. They, just like enough humans do already, imitate the results of "having an adventure of the soul".
> "Can computers generate music?" (Yes) and "Can computers generate music that is enjoyable to listen to?" (Not yet)
And we're talking about the question "should they?", which science can't even attempt to answer. "Play from your heart", and all that; not even best-selling artists pumping out mediocrity are above that criticism, even when they do it according to the best of their ability and conscience, and even when it makes people "happy".
We chose to work on music because we want to continue to push the boundaries of generative models.
Alternatively, here, we are still witnessing art. The artist, as ever, is human: the scientists who pieced together these techniques. Theirs is the voice, if only humans can have a voice, that we hear in the work.
They are not semantically empty: they are absorbed, semantically, in the domain of the computer scientist who, through no fault of their own, could never sing before now.
So a computer communicating the aspects of its life based on the facts and experiences it has been fed is any less valid?
If turtles spontaneously developed human-level intelligence and created music, would it "miss the point of music" for not conveying human experinces?
What does that even mean? As a counter-point I listen to heavy bass music with zero lyrics. The production value is the most important thing for me and I would 100% listen to AI generated music.
Finally, can we stop treating ever single piece of work by neural networks as a "failure" because it isn't GAI? Just because it doesn't "say something about the human experience", doesn't make it bad engineering. It's hilarious how as soon as there's some new AI work done everyone starts wailing, "where's the humanity!"
Lay-people think AI refers to ALife.
Most of the talking heads would be immediately satisfied—giving none of these complaints—if they were shown an "AI" program that responds to stimuli by entering emotional states, and which learns to associate stimuli with the emotional states it has been in in the past, such that those stimuli will then become triggers for those states, and for memories associated with those states.
Such an agent wouldn't even need to use ML techniques, necessarily. It'd just need to be a high-concept tamagotchi that can respond to operant conditioning. That would already be an advance over the state of the art.
But, AFAIK, nobody's really working on ALife in the sense of "making an individual agent with a complex-enough internal model that it can statefully respond to you the way a pet does." ALife is only really studied at the very low level (C. Elegans connectome simulation) or the very high level (sociological/economic simulations using simple goal-driven agents); nobody's really working in the space "in between." (Except for the people trying to make chat bots seem friendlier, but they're mostly trying to fake it, rather than creating actual persistence-of-memory.)
I wonder why nobody's interested in medium-scale ALife research these days? It used to be a hot topic, back when it was conflated with robotics under the banner of "embodied cognition."
Now, is A[rtificial] Life the correct term to use here? I feel it isn't - I'd expect ALife to be more concerned with implementing simulacra of bacteria or worms in silico, not with reasoning or emotions.
One might say that ALife is to an event loop as AI is to a one-time query-response. AI can evaluate, but you need an ALife system in order to "think" in a continuous way.
There's really no sense in which an ALife researcher cares about recreating a full-fidelity model of biology in silico; the point is to specifically study the thinking and decision process of real agents, and figure out how to model those, in a way that the model makes the same series of decisions the real agent does in the same situations (and, therefore, must also be keeping and updating analogous internal state to the kind the real agent keeps.)
Some of those models are attempts to recreate real brains/nervous systems, but these models aren't fundamentally biological. A "low level" connectome simulation doesn't contain any model of cellular inflammatory response, cellular waste and its clearance, etc. It's basically just a brain-as-actor-model with neurons as stateful processes and electrochemical signals as messages.
An ALife researcher cares about as much about biology below the level of intracellular pharmacodynamics (sodium channels et al), as a race-car-chassis engineer cares about physics below the level of fluid dynamics. They don't need to go any lower, because they've found an encapsulating abstraction that makes all the predictions they're interested in making, without needing any lower-level information.
Just think about how GANs were viewed when they were first published. The common sentiment was that it was as an interesting "research contribution" that could never live up to the hype. However, the promise behind it inspired people to continue to work on it and now we're able to produce realistic human faces that humans can't tell are fake.
I actually wanted to keep listening to this one: https://jukebox.openai.com/?song=799583581
And this wasn't bad, sounds like something you'd see from some 1940s-era newsreel: https://jukebox.openai.com/?song=799583728
https://towardsdatascience.com/the-most-important-supreme-co...