Wow, that was actually impressive. A few of those were hard to distiguish from human speakers. I love how they included breathing and mouth-sounds as well, really brings it to life.
They say they doesn't charge for royalties for using generated audio. This could be huge for accessibility and alternative forms of media consumption for articles and online media. Generate once, distribute to all.
Shameless plug: check out Tropo[1]: you can interface with SIP.
One thing to keep in mind regarding speech to text: if you need accuracy, it helps immensely if you can guide the recognizer with a list of expected answers. Un-hinted transcription will do its best to recognize words but without AI to understand the context, there are still quite a few possibilities for mis-recognition. More info about these two here[2].
I'm a native Portuguese speaker and I can say that this is far from how people speak Portuguese. It sounds like pretty much other TTS solutions.
However, Spanish and English are awesome and really sound like a person. This is great!
Edit: After playing a little bit with it, it does sound better than anything I have seen before even in Portuguese. I believe that the weirdness is just for the standard/demo phrase. When I tried some custom phrases (curse words, mainly) it does sound very natural.
Given that this supports Speech Synthesis Markup Language [0], I wonder what the impact will be on positions that talk for a living. While this is unlikely to replace voice actors for games/cartoons, I wonder if public service announcers and such will be replaced by this robot.
One thing I always still notice about these "lifelike" speech models is that they still have random pitch variation that wouldn't be present from a real speaker—sort of a "warbling", similar to audio heavily compressed by some cellular realtime audio codecs.
I assume this is due to random variation in the pitches the speakers in the training data used to say given phonemes. Would this, then, imply that speech models for tonal languages like Mandarin don't share this problem, since people are speaking with prescribed pitches more often?
Re: the checker-boarding artifacts, that sounds interestingly like deconvolutional NNs treat their samples, at each deconv layer, as independent, rather than taking the Rule of Addition of probability-weights (P(A ∪ B) = P(A) + P(B) - P(A ∩ B)) into account. (Would a model that did do this then be a "Bayesian Deconvolutional Neural Network"? Does that even make sense?)
I doubt this would result in audible artifacts at the audio level, though. The visual checkerboarding appears at the 1px detail-level of the output image—this would translate more to minor fluctuations along an audio sample's Nyquist frequency, which is effectively undetectable if the output sampling rate is of the usual kind (44.1kHz).
I always wonder why they don't program a second pass over the output with a model, say, trained on radio conversations. That second model just for audio 'quality'.
Additionally when we speak we don't just 'replay' words; we add emotions and so on. Reminds me of this video [1] analysing how the actor Anthony Hopkins converts his lines to speech for his scenes.
The spoken word has so many nuances. Cadence, inflexion, tone, and then the facial performances are never captured by pure text to speech. You'd almost have to be able to set boundaries for seriousness, playfulness, innuendo, indifference, sarcasm etc because none of that is conveyed in the literal text without additional descriptions or contextual analysis of the conversation.
It'd be interesting to take the Joint Many-Tasks approach here, training some of an RNN's layers on text-to-speech, and then another set of layers on sentiment analysis of speech-audio, where the error of the sentiment layer can backpropagate into the TTS layer.
I guess you could do something like vocaloid does to better "shape" the sounds of each word. I guess its pretty tough to do that automatically though, since words by themselves don't convey the emotions that they should be portrayed with.
I mean, a word or phrase could very much change meaning depending on how its spoken (excitement, disgust, fear, sarcasm etc) and even if the context is available in the text (as is often the case), its rather difficult to machine-detect, at least, for now.
Wouldn't we describe what TTS systems do as more of a speech impediment? Sure they use a particular accent, usually that of a standard prestige dialect. But it's not the accent that's weird.
If you grab the training set for CMU-Sphinx, you'll see that the voice actors providing base input are trained to speak that way.
At a secondary level, I imagine there's a sort of pitch trajectory for a sentence, and if a word or syllable ends too far off that trajectory, the return to the pitch trajectory is fast enough to be easily detectable.
I don't know about tonal languages. But you're definitely right about the artifacts of concatenative speech synthesis.
It's worth noting that purely synthetic speech, while lacking the superficial lifelike quality, doesn't have these little inconsistencies. Perhaps that's one reason why many blind people, particularly power users, prefer pure synthetic text-to-speech engines, such as ETI-Eloquence (commercial) and eSpeak (open source). These systems especially sound better at high speeds than the newer "lifelike" ones.
Just for fun, here's a side-by-side comparison between a concatenative system, IVONA (acquired by Amazon):
Those phrases sound very natural indeed. But are they carefully selected demo phrases, for which the system might have been tuned? That's always a danger with marketing demos. So I have two questions:
1. How does that engine sound when reading arbitrary text at high speed?
2. Can I run it on my own computer, so I can have low latency in my screen reader? (That, of course, is a problem with Amazon Polly as well.)
Could we drop “Lifelike” from the title? It's marketing puffery (and untrue, it sounded very robotic to me to the point I was convinced I'd heard the wrong audio file). The actual story is that it's Amazon's text-to-speech-as-a-service offering, not that it's particularly innovative in terms of sounding good.
I agree that the voice sounds pretty bad, but I think the 'lifelike' part comes from the intelligence part. I.E WA -> Washington, and 75F -> 75 Fahrenheit
Text-to-speech systems have been expanding abbreviations for decades, and it always backfires in some cases. For example, DECtalk would expand "Sun" to "Sunday" regardless of context, with the humorous result that blind people reading tech news in the 90s would often hear about Sunday Microsystems.
Edit: Yes, that example was unfair, because it's from the 90s (and actually, DECtalk was largely unchanged since the 80s, until it was ruined in the late 90s). Here's a somewhat more recent one: With the ETI-Eloquence engine, which was last updated in 2002, the string "2 Marketing" (which one might find, say, in a Wikipedia table of contents) is expanded to "March twond keting".
One blind guy I know opined in the late 90s that it's better to let the brain do the interpreting. But I understand that expansion of abbreviations makes TTS more attractive for mainstream applications.
I was looking around for samples, and this is the only sample I have heard so far. I think that sounds pretty good. The emotional tone is off, but it is a lot better than say, Siri (which I understand was designed to sound robotic to highlight the tech). But I'm also the guy who has had my MBP announce the hour and it has freaked out coworkers.
I don't know if this is crossing into uncanny valley territory. I think some of this is not being used to it.
Google Map's driving voice sounds subtly better than that voice sample. I'm looking forward to seeing if a similar service is coming for GCP.
> the 'lifelike' part comes from the intelligence part. I.E WA -> Washington, and 75F -> 75 Fahrenheit
This would drive me nuts, as there's no easy way to predict what sort of transformations it would make. Basically, if it's not near perfect, it's arguably worse than no intelligence at all.
With our PBX stuff, I know it's dumb, so I'm already accustomed to giving it something I know it will read correctly, like "domain dot com".
Sooo cynical. This is way better than most TTS systems, including Google's. The only one I've ever heard that is better is DeepWave, and that suffers from lots of background noise.
Do you mean WaveNet? The noisiness is because it uses 8bit (mu-law) audio. 8bit is because WaveNet is computationally extremely expensive. It's pretty much proof of concept at this stage, not production ready TTS.
It's more like the opposite of being innovative (unimaginative?)... They bring a product that could previously achieve that locally to a remote server and the quality is not any better.... and they want to charge you by the character... no thanks Amazon... hopefully it's free with Prime.
English speakers sound brain dead to be honest. This sounds no different from pre-NN attempts, only difference this time is it's behind an Amazon paywall.
The thing these voices always miss is context. We need about 700 variations of every textual sentence that all depend on the mood or goal of the conversation and modulate pitch, volume, speed, word choice, etc. appropriately.
This experiment does cover the "generic computer voice" role quite well, but customers don't want that on e.g. their website. I would want a "I'm a professional designer" voice. So there will always be markets for more appropriate voices [*]. But this one does sound very nice at what it does.
The same will happen with robots, btw. Once we get a generic humanoid robot that can do everything, we will still seek aesthetic variations and employ remote body actors.
How is directly linking a .mp3 the best way to preview their technology? You'd think there would be a much more elegant solution to showcase Polly on their site.
The demo files in the article sound pretty decent, but this sample is pretty bad. It sounds pretty robotic by itself (lots of artifacts, odd pitches, no pauses), but saying "during the one thousand nine hundred and seventies" is the real kicker.
I really like text2speech because I really like the computer to read articles to me while I'm doing something else, but I won't use any internet service for that just to get a little bit more quality in comperison to Apples "Alex" or MeryTTS http://mary.dfki.de/
There's a lot of negative reaction to Polly. Besides WaveNet (which is not available as an API-- and really slow), what are better alternatives? Or is the reaction just to Amazon's marketing speak using "lifelike" and "deep learning"? I find most of the voices to be okay, but Joanna is pretty good and better then anything I've heard besides WaveNet.
Also, with Amazon's FPGA investment how long before it implements WaveNet using FPGA's to reduce the speech generation time to something that is realtime?
Why would Google use FPGAs over its own (faster) TPUs? Chances are Google either doesn't have enough TPUs (it may be waiting on the next generation to go massive scale with them), or Wavenet is still too computationally intensive even for TPUs.
I am sure Google is looking at WaveNet & TPUs, I was just thinking Amazon would look at using its new massive FPGAs to implement WaveNet. I have no idea if the FPGAs would be fast enough.
105 comments
[ 2.7 ms ] story [ 162 ms ] threadFor a service claiming to be "lifelike", Joey has totally phrased that question as a statement.
[1] https://deepmind.com/blog/wavenet-generative-model-raw-audio...
https://console.aws.amazon.com/polly/home/SynthesizeSpeech
One thing to keep in mind regarding speech to text: if you need accuracy, it helps immensely if you can guide the recognizer with a list of expected answers. Un-hinted transcription will do its best to recognize words but without AI to understand the context, there are still quite a few possibilities for mis-recognition. More info about these two here[2].
[1] https://www.tropo.com
[2] https://www.tropo.com/docs/voice/transcription-vs-speech-rec...
(Full disclaimer: I'm in the Tropo BU)
However, Spanish and English are awesome and really sound like a person. This is great!
Edit: After playing a little bit with it, it does sound better than anything I have seen before even in Portuguese. I believe that the weirdness is just for the standard/demo phrase. When I tried some custom phrases (curse words, mainly) it does sound very natural.
https://www.w3.org/TR/speech-synthesis/
I assume this is due to random variation in the pitches the speakers in the training data used to say given phonemes. Would this, then, imply that speech models for tonal languages like Mandarin don't share this problem, since people are speaking with prescribed pitches more often?
Here's a good writeup: http://distill.pub/2016/deconv-checkerboard/
I doubt this would result in audible artifacts at the audio level, though. The visual checkerboarding appears at the 1px detail-level of the output image—this would translate more to minor fluctuations along an audio sample's Nyquist frequency, which is effectively undetectable if the output sampling rate is of the usual kind (44.1kHz).
[1] https://www.youtube.com/watch?v=4kSGkGKwp9U
I mean, a word or phrase could very much change meaning depending on how its spoken (excitement, disgust, fear, sarcasm etc) and even if the context is available in the text (as is often the case), its rather difficult to machine-detect, at least, for now.
Example (click on the blue speaker icon): http://dict.baidu.com/s?wd=%E7%88%B1%E5%B1%8B%E5%8F%8A%E4%B9...
At a secondary level, I imagine there's a sort of pitch trajectory for a sentence, and if a word or syllable ends too far off that trajectory, the return to the pitch trajectory is fast enough to be easily detectable.
It's worth noting that purely synthetic speech, while lacking the superficial lifelike quality, doesn't have these little inconsistencies. Perhaps that's one reason why many blind people, particularly power users, prefer pure synthetic text-to-speech engines, such as ETI-Eloquence (commercial) and eSpeak (open source). These systems especially sound better at high speeds than the newer "lifelike" ones.
Just for fun, here's a side-by-side comparison between a concatenative system, IVONA (acquired by Amazon):
http://mwcampbell.us/tmp/derefr_comment_ivona.mp3
And a purely synthetic system, Eloquence:
http://mwcampbell.us/tmp/derefr_comment_eloquence.mp3
The latter is the text-to-speech engine I use every day, and that clip was synthesized at the speaking rate I normally use.
Edit: It appears the word I was looking for when I said "purely synthetic" is "parametric".
• https://storage.googleapis.com/deepmind-media/pixie/us-engli...
• https://storage.googleapis.com/deepmind-media/pixie/us-engli...
1. How does that engine sound when reading arbitrary text at high speed?
2. Can I run it on my own computer, so I can have low latency in my screen reader? (That, of course, is a problem with Amazon Polly as well.)
I don't see (or hear?) the advance here...
Edit: Yes, that example was unfair, because it's from the 90s (and actually, DECtalk was largely unchanged since the 80s, until it was ruined in the late 90s). Here's a somewhat more recent one: With the ETI-Eloquence engine, which was last updated in 2002, the string "2 Marketing" (which one might find, say, in a Wikipedia table of contents) is expanded to "March twond keting".
Someone linked another example here:
https://soundcloud.com/zack-bloom/amazons-new-text-to-speech...
And it pronounces 1970s (the decade) as one-thousand-nine-hundred-seven-ths.
I don't know if this is crossing into uncanny valley territory. I think some of this is not being used to it.
Google Map's driving voice sounds subtly better than that voice sample. I'm looking forward to seeing if a similar service is coming for GCP.
This would drive me nuts, as there's no easy way to predict what sort of transformations it would make. Basically, if it's not near perfect, it's arguably worse than no intelligence at all.
With our PBX stuff, I know it's dumb, so I'm already accustomed to giving it something I know it will read correctly, like "domain dot com".
I believe that's what the Lexicons are for within the service.
I read the post I was replying to as some sort of undefined, built-in, set of intelligent transformations.
For example, 75F-> 75 Fahrenheit would require that you make a Lexicon entry for every value...1F, 2F, 3F, etc. That's not really intelligent :)
Polly 1: https://d0.awsstatic.com/product-marketing/Polly/HelloEnglis...
Polly 2: https://d0.awsstatic.com/product-marketing/Polly/HelloEnglis...
WaveNet 1: https://storage.googleapis.com/deepmind-media/pixie/us-engli...
WaveNet 2: https://storage.googleapis.com/deepmind-media/pixie/us-engli...
There just isn't a comparison -- Polly sounds way more robotic.
WaveNet is Google's, and I'm saying that this is significantly worse.
This experiment does cover the "generic computer voice" role quite well, but customers don't want that on e.g. their website. I would want a "I'm a professional designer" voice. So there will always be markets for more appropriate voices [*]. But this one does sound very nice at what it does.
The same will happen with robots, btw. Once we get a generic humanoid robot that can do everything, we will still seek aesthetic variations and employ remote body actors.
> The size of the input text can be up to 1500 billed characters (3000 total characters). SSML tags are not counted as billed characters. [1]
Not sure where you got 1000 from. As to the why - no. It doesn't seem like too much. 2 minutes of speech maybe in there?
[1]http://docs.aws.amazon.com/polly/latest/dg/limits.html
This is in comparison with Adobe's tech where it samples a real human voice and allows speech to be inserted (http://arstechnica.com/information-technology/2016/11/adobe-...).
To play around with it: https://console.aws.amazon.com/polly/home/SynthesizeSpeech
Also, with Amazon's FPGA investment how long before it implements WaveNet using FPGA's to reduce the speech generation time to something that is realtime?