Maybe not yet, but I am sure there will be speaker cloning extension soon, like photo and painting style transfer.
Edit: It would seem they only need 25 hours of audio to train the model, so it would probably be easy to find or make alternate datasets.
> We train Tacotron on an internal North American English dataset, which contains about 24.6 hours of speech data spoken by a professional female speaker
It was pointed out the last time I posted this that the demo is "fake" at least in as much as the inserted words aren't being synthesized but rather copy-pasted with existing intonation intact.
It's not really style transfer, but for a new speaker model, you just need to train each speaker with a dataset of 25 hours audio with time matched accurate transcriptions.
In the case of David Attenborough, I'm pretty sure that amount of data is available with subtitled BBC content.
Oh I doubt that will be necessary. I bet you could find a useful "feature vector" by repeating the process on about 10-20 new voices - then extract a set of words (let's pessimistically say 100) that activate these features the most - then have a person retrain with 5 examples of each of these 100 words (an hour, at the most).
I honestly wouldn't be surprised if you could find a really feature rich sentence (say 5 words long) that you could use to crack pretty much all voice-activated biometric password systems.
My understanding is that while a number of people have tried, we currently have no particularly viable approaches for 'style transfer' analogues on sequence data, including text and sound/speech. Samples from past work are pretty bad. But what you could possibly do is roundtrip audio to text and then text (with conditioning like speaker metadata) to speech, both of which are doable. But directly transforming, not yet.
These new models are replacing almost everything that's been developed for TTS in the last 30 years with neural nets. So you don't need to study TTS anymore. Study neural nets and then read this paper and you will understand the state of the art in TTS.
This is complete nonsense. You need to understand the problem you are addressing even if you plan to use deep learning methods for them.
Saying you don't need to know anything about TTS is like developing a self-driving car and saying you don't need to know anything about the rules of the road. For one thing, how are you supposed to know when you got it wrong? And what things it is important to get right?
Fyi they added newer, (imo) better quality voices since alex debuted in 10.5. My personal favorite is Kate (english UK voice) and Tessa (english - south africa)
The stress and intonation examples are what puts this on the far side of the uncanny valley. I've not heard computer speech like this, where the voice seems understand the idea underneath the words.
"WaveNet (van den Oord et al., 2016) is a powerful generative model of audio. It works well for TTS,
but is slow due to its sample-level autoregressive nature. It also requires conditioning on linguistic
features from an existing TTS frontend, and thus is not end-to-end: it only replaces the vocoder and
acoustic model. Another recently-developed neural model is DeepVoice (Arik et al., 2017), which
replaces every component in a typical TTS pipeline by a corresponding neural network. However,
each component is independently trained, and it’s nontrivial to change the system to train in an
end-to-end fashion."
It is without doubt very impressive, but on longer texts the lack of understanding will be apparent. Wrong semantic stresses, incorrect pronunciation of homographs, maybe inconsistent pronunciation of names, and so on.
Perfect speech synthesis from unannotated text will require much more complex system. It will need database of common knowledge in some form, ability to relate text to that database, ability to retain and use this semantic information to guide synthesis. That is, it should be a step closer to artificial general intelligence.
I'd say even in these samples there's still some errors. The intonation on "brown" in "quick brown fox" is still wrong. "AllHipHop" is not pronounced as a human would, though I'll grant you that's a made-up word so I wouldn't even expect a non-native human speaker to necessarily get that one right.
Still, the quality is such that I'm only observing this in the spirit of scientific analysis. It's some good quality.
I wonder whether it would have been more natural if it was written as "quick, brown fox" as would be customary English. Usually when adjectives don't have commas between them it's because the last adjective is coupled to the noun.
How difficult would it be to duplicate these results with TensorFlow? Would something like this require more than the building blocks that TF and other toolkits provide? I have zero experience with machine learning, so I'm just curious.
They mention implementing the model in TensorFlow, so it should absolutely be possible. (Unless they are using a special Google-only version.) One thing that might make reproduction difficult is that they don't describe their initialization procedure. But maybe it just doesn't matter.
1.Machine learning is pretty data dependent, and make those datasets are very expensive. Google is not likely to give them away for free, because it is their competitive advantage.
2.The infrastructure to train those models are hard to get outside of Google. Pretty sure it is 10s or 100s of GPUs, with Infinity Band connected PS server, running for days and weeks.
Even with source code published, people will still have to scratch their head to duplicate Google's performance. Until the day, some equivalent organization as GNU that democratize data access to the public and some mighty algorithm being discovered dramatically reduced the computational requirement for training those models, Google succeeds by just being Google is unlikely going to change.
Right, even a good diphone voice needs lots of data. And I noticed they trained it with the existing Google Home voice actress, from whom they must already have many, many hours of recordings. I was mostly asking about the model itself; whether you could download TensorFlow and put one together based on this paper alone.
I see your points. But it is related. Even if u get what you think the paper describes, it is hard to know whether you did it right or not, because you cannot replicate the result easily. This happens in a lot of CV papers already, where people reimplement the model, but it never get as good as the paper demonstrated
But, you have a very good idea. Since it is Google Home, will it be possible that some people just buy hundreds of them, and infinitely ask them question to gather the training data? That will be interesting to see.
1. If you can pay for around 24.6 hours of VA speech data, you can get enough data to run this process with the same quality that Google presented. (that's from the "Experiments section") Not cheap (definitely not free, especially considering the amount of quality control you have to apply), but not expensive either.
2. You can rent out a 96GB GDDR5 GPU instance from Google's cloud for pretty cheap. (https://cloud.google.com/compute/docs/gpus/) I don't think you need anything more powerful than that (but feel free to prove me wrong).
I think your last paragraph is totally misguided/uninformed. You can download models for cheap/free (for non-commercial/edu use) from UPenn (https://www.ldc.upenn.edu/language-resources/data/obtaining). People don't give away models for free with 0 strings attached because they're a pain to make.
And if you want something you can run on a home computer for cheap/free, you can try DeepSpeech: https://github.com/mozilla/DeepSpeech. All you need is an Nvidia GPU.
1.What is your estimate then? Hundreds Dollars? Thousands? 10 Thousands? Surely it might fall into the later two category, since paying a professional speaker to sit and work for 26.4 hours is already over 1000 dollars, if you assume 40$ per hour wage.
2.96GB GDDR5 instances on GCE costs 4166.4 dollars per month. Though it is within affordable range, but definitely not CHEAP. I don't know whether this is powerful or not, but Google used 96 GPUs for their GNMT work. Thus, I don't think I have the confidence to say a 4-GPU machine is all you need, and it will surely cause much more if you go beyond that.
How about feeding it several audiobooks read by a single narrator, coupled with the books in text? Cost would be < $100. There could be legal problems if you tried to sell the resulting voice, but as proof of concept, wouldn't this work?
The model is smart enough to not mix them together in the same sample. Different samples would have different voices, unless you put in a voice identifier as input to the model. Then you can control which voice to generate with.
From what I understand looking through the paper, this is only a research paper, without any source code. So right now it's not possible to say what the hardware requirement might be.
Feels like we're getting breakthroughs every few months now. How close are we to the point yet that a mobile phone can generate speech in close to realtime that is reasonably pleasant to listen to for long periods?
I just think there are huge opportunities for new types of UI once it works. Any time I'm out and about with headphones, I want something like a custom radio station, mixing up my music and podcasts, dictating messages and highlights from social media or news sites I like. I'd even accept short adverts if it was genuinely relevant and location aware. Even when I'm not travelling, I suspect it'd be a cool interface to have at work. If something can check in from time to time between music tracks, giving me a quick run down of anything interesting that's happening (even plugins for the build server etc), it will cut down on my temptation to open it up in a browser tab.
I've wanted to build this app just for myself for years, but it's only very recently that it seems the state of the art is close enough for it to be palatable to users.
Huh? I would think, if anything, it would go down if, as an alternative to pirating an audio book, one can download the text from Gutenberg.org or pirate the _text_, and generate a decent audio book from it.
True, technically it's not "audiobook piracy", but I used that as shorthand for recording and distributing an audio version of a written book, which courts will surely find to be a form of piracy.
The reason is, the only speech data that is available in high quality and large quantities is in audio form. There has been research for decades on measuring or estimating the properties of the speech organ (oral cavity geometry etc), but that is largely an unsolved model inversion problem. Existing physical speech synthesis models are pretty crude and sound quite robotic.
ML is strongly related to solving inverse problems but it does not tend to be emphasised enough, in my opinion.
In reality, speech organs (a relatively low dimensional system) generate high dimensional data (speech). This is an example of the so called curse of dimensionality.
What ML does is to invert this process and create a model of the generation process.
My favorite example is quantum mechanics. Physicists made lot of measurements and created a model (quantum mechanics) that reproduces those measurements (and even works on test data, not only on training data).
This is an example of a solution of an inverse problem. ML does the same.
So in one way or an other, every working TTS has a model that accurately describes the speech organs and how the brain uses them to create speech from text.
Unless all these fancy new text to speech engines are released as actual software instead of cloud services they're worthless. And we all know that isn't going to happen.
I'd really love this to mature completely and applied to Kindle Whispersync books. I've found that some readers of audiobooks make the book hard to comprehend or even distracting, especially if the sex of the person does not match the author or even remotely the sound of the author who I may be very familiar with from video interviews and what not. Ideally every author would be a great reader and create recordings for their own content, but if some synthetic means could achieve an approximate result that would be fantastic. Even better people could choose from a flavor of voices and settle upon the one that works for them.
86 comments
[ 3.4 ms ] story [ 156 ms ] threadEdit: It would seem they only need 25 hours of audio to train the model, so it would probably be easy to find or make alternate datasets.
> We train Tacotron on an internal North American English dataset, which contains about 24.6 hours of speech data spoken by a professional female speaker
[1] http://www.washingtonpost.com/wp-srv/national/dotmil/arkin02...
In the case of David Attenborough, I'm pretty sure that amount of data is available with subtitled BBC content.
I honestly wouldn't be surprised if you could find a really feature rich sentence (say 5 words long) that you could use to crack pretty much all voice-activated biometric password systems.
[0] https://deepmind.com/blog/wavenet-generative-model-raw-audio... [1] http://www.festvox.org/
Saying you don't need to know anything about TTS is like developing a self-driving car and saying you don't need to know anything about the rules of the road. For one thing, how are you supposed to know when you got it wrong? And what things it is important to get right?
http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf
IMO, it actually performs pretty robustly on these examples. Is Apple still using just diphones, or are they post-processing in some way?
[0]: https://en.wikipedia.org/wiki/PlainTalk#Text-to-speech_in_Ma...
I hope we will have decent TTS for linux too.
[0]https://deepmind.com/blog/wavenet-generative-model-raw-audio...
"WaveNet (van den Oord et al., 2016) is a powerful generative model of audio. It works well for TTS, but is slow due to its sample-level autoregressive nature. It also requires conditioning on linguistic features from an existing TTS frontend, and thus is not end-to-end: it only replaces the vocoder and acoustic model. Another recently-developed neural model is DeepVoice (Arik et al., 2017), which replaces every component in a typical TTS pipeline by a corresponding neural network. However, each component is independently trained, and it’s nontrivial to change the system to train in an end-to-end fashion."
Perfect speech synthesis from unannotated text will require much more complex system. It will need database of common knowledge in some form, ability to relate text to that database, ability to retain and use this semantic information to guide synthesis. That is, it should be a step closer to artificial general intelligence.
Still, the quality is such that I'm only observing this in the spirit of scientific analysis. It's some good quality.
But damn, the rest of it is astounding
If anything, it sounds like "quiley".
How difficult would it be to duplicate these results with TensorFlow? Would something like this require more than the building blocks that TF and other toolkits provide? I have zero experience with machine learning, so I'm just curious.
Very difficult:
1.Machine learning is pretty data dependent, and make those datasets are very expensive. Google is not likely to give them away for free, because it is their competitive advantage.
2.The infrastructure to train those models are hard to get outside of Google. Pretty sure it is 10s or 100s of GPUs, with Infinity Band connected PS server, running for days and weeks.
Even with source code published, people will still have to scratch their head to duplicate Google's performance. Until the day, some equivalent organization as GNU that democratize data access to the public and some mighty algorithm being discovered dramatically reduced the computational requirement for training those models, Google succeeds by just being Google is unlikely going to change.
But, you have a very good idea. Since it is Google Home, will it be possible that some people just buy hundreds of them, and infinitely ask them question to gather the training data? That will be interesting to see.
2. You can rent out a 96GB GDDR5 GPU instance from Google's cloud for pretty cheap. (https://cloud.google.com/compute/docs/gpus/) I don't think you need anything more powerful than that (but feel free to prove me wrong).
I think your last paragraph is totally misguided/uninformed. You can download models for cheap/free (for non-commercial/edu use) from UPenn (https://www.ldc.upenn.edu/language-resources/data/obtaining). People don't give away models for free with 0 strings attached because they're a pain to make.
And if you want something you can run on a home computer for cheap/free, you can try DeepSpeech: https://github.com/mozilla/DeepSpeech. All you need is an Nvidia GPU.
2.96GB GDDR5 instances on GCE costs 4166.4 dollars per month. Though it is within affordable range, but definitely not CHEAP. I don't know whether this is powerful or not, but Google used 96 GPUs for their GNMT work. Thus, I don't think I have the confidence to say a 4-GPU machine is all you need, and it will surely cause much more if you go beyond that.
So why is it on Github?
Still no source code...
There will be a tensorflow implementation tomorrow though.
I have my reasons.
* Is this open source?
* What hardware would one need to run this?
* If specialist hardware is required (e.g. reasonably high end GPU) would that simply be for building and training the system or its operation?
I've wanted to build this app just for myself for years, but it's only very recently that it seems the state of the art is close enough for it to be palatable to users.
Would it be easier for a neural network to "control" an emulated throat/mouth and train against the sound output?
Emitting and post-processing waveforms entirely in neural networks seems like putting a lot of responsibilities into neural networks.
Could you provide some links/publications please?
ML is strongly related to solving inverse problems but it does not tend to be emphasised enough, in my opinion.
In reality, speech organs (a relatively low dimensional system) generate high dimensional data (speech). This is an example of the so called curse of dimensionality.
What ML does is to invert this process and create a model of the generation process.
My favorite example is quantum mechanics. Physicists made lot of measurements and created a model (quantum mechanics) that reproduces those measurements (and even works on test data, not only on training data).
This is an example of a solution of an inverse problem. ML does the same.
So in one way or an other, every working TTS has a model that accurately describes the speech organs and how the brain uses them to create speech from text.