I wonder why Google is doing it like this. Sending an audio file of your voice over the wire seems like unnecessary overhead. From an API perspective I'd much rather have client-side speech-to-text built into the browser (accessible via a JavaScript API). I think they're already doing STT client-side in Android, so what's the hold-up with embedding it in Chrome and letting web developers go nuts with voice-enabling their web apps?
I'm not familiar with Android, so I'm not sure about the capabilities of its client side speech recognition algorithms, but the short answer is that competent speech to text is currently a very, very intensive problem.
SR is roughly divided into acoustic and language modeling. The acoustic model proposes words that might have been said given some chunk of speech and the language model tells you what the most likely actual word is given what's been said.
The acoustic model can be solved in a large number of ways — though production technologies use very large hidden Markov models — but decoding a word sequence from speech might scale like O(knm^2) with n being the size of your vocabulary (often large), m the complexity of the acoustic model (# of phonemes modelled, perhaps) and k the number of acoustic frames. The n at the very least can be parallelized (embarrassingly), but the m and k cannot.
The language model involves a search through an exponential search space of orderings of words in the vocabulary (n^l choices, but l is also unknown). Anything sophisticated also will have an incredibly large (in memory) model as it has to have parameters across words, pairs of words, triples of words, grammatical categories, topics, etc. etc.
Solving both of these problems well simultaneously is not a task for a consumer computer. Speedy algorithms with small vocabularies and simple models exist and are implemented (Dragon Naturally Speaking, for instance) but Google didn't go and record a million hours of GOOG-411 to produce Dragon's technology over again.
---
Finally, there's a lot of work on front end signal processing in SR. Before you get into acoustic and language modeling, you often transform your input into another representation (often spectral components from sliding 10ms frames). A growing camp in SR research involves finding sparse front end representations of speech though. If the client-side software is capable of quickly computing a sparse representation of the speech, that could dramatically reduce the latency and bandwidth issues.
By the way, the reason Google is willing to spend this computational effort for your convenience is probably the same as GOOG-411. They are definitely recording every translation they do in order to use as a huge training corpus later.
We don't know what algorithm they are using or how computationally intensive it is. It may be the case that it would choke anything but the most powerful modern desktops/laptops; they can afford to throw 32 cores at the problem for a fraction of a second where you can't, or it may be backing to an arbitrarily large pre-computed lookup table of some type stored in memory on these machines. I doubt it's that bad, I just made a number up, but it could be bad. And cell phones may not be able to do anything with it at all. Using a server gives them the freedom to use more power, automatically collect the results for more learning, and not need to worry about compatibility. (Or reverse engineering.)
To throw out another interesting number, I often hear people complaining about finding ways to fit hundreds of gigabytes of language models into memory in order to make search complete before heat death.
That's really just not a cell phone level of computation yet.
Android STT isn't client side, that's why you get a "connection error" message if you turn data off and try it. STT is a problem best solved with big iron.
There's actually two main reasons, related to each other:
In order to do the kind of open domain recognition google (and others) do here, you need data. Lots of data. That implies that the models themselves are very large, and (generally) require more CPU to recognize with. You can't do that on a mobile device, mainly because of the space issue, but fancy new algorithms probably consume more CPU than even the latest mobile hw can manage[1]
The second, related issue is that, again, in order to do this kind of open domain recognition, you need to constantly improve the models[2]. Even a moderately sized set of models would be a pain in the rear to send back up to every android device using the system.
[1] You can typically do the front-end feature processing (and adaptation) on the device, and most vendor's solutions end up doing a bit of that.
[2] Even for speaker dependent modelling (dictation), you'd probably want to associate a key with the model and adapt that model in the cloud rather than sending model updates back to the device because of the space and transmit times.
Might be worth mentioning that the Google Translate app for IOS already had this feature for a while. It actually works surprisingly well in all the languages I have tried (even though I speak some of them with a horrible accent).
I suspect they just added an interface for Chrome to an existing back-end.
Update: I just noticed that this is actually an implementation of the HTML Speech Input API (http://lists.w3.org/Archives/Public/public-xg-htmlspeech/201...), so technically this could appear on any site. This makes things actually a lot more interesting :)
What's interesting to me about the Speech Input API is that most browser vendors don't have access to the underlying technology to make the api useful. Microsoft has the tech, and Apple has it a little bit, but certainly Mozilla and Opera do not. They might be able to afford providing it through 3rd party access, but any one smaller than that (read: community browsers) aren't going to be able to implement this.
It's the first HTML standard that I can think of where this is true.
14 comments
[ 11.8 ms ] story [ 223 ms ] threadSR is roughly divided into acoustic and language modeling. The acoustic model proposes words that might have been said given some chunk of speech and the language model tells you what the most likely actual word is given what's been said.
The acoustic model can be solved in a large number of ways — though production technologies use very large hidden Markov models — but decoding a word sequence from speech might scale like O(knm^2) with n being the size of your vocabulary (often large), m the complexity of the acoustic model (# of phonemes modelled, perhaps) and k the number of acoustic frames. The n at the very least can be parallelized (embarrassingly), but the m and k cannot.
The language model involves a search through an exponential search space of orderings of words in the vocabulary (n^l choices, but l is also unknown). Anything sophisticated also will have an incredibly large (in memory) model as it has to have parameters across words, pairs of words, triples of words, grammatical categories, topics, etc. etc.
Solving both of these problems well simultaneously is not a task for a consumer computer. Speedy algorithms with small vocabularies and simple models exist and are implemented (Dragon Naturally Speaking, for instance) but Google didn't go and record a million hours of GOOG-411 to produce Dragon's technology over again.
---
Finally, there's a lot of work on front end signal processing in SR. Before you get into acoustic and language modeling, you often transform your input into another representation (often spectral components from sliding 10ms frames). A growing camp in SR research involves finding sparse front end representations of speech though. If the client-side software is capable of quickly computing a sparse representation of the speech, that could dramatically reduce the latency and bandwidth issues.
The best data is more data.
That's really just not a cell phone level of computation yet.
In order to do the kind of open domain recognition google (and others) do here, you need data. Lots of data. That implies that the models themselves are very large, and (generally) require more CPU to recognize with. You can't do that on a mobile device, mainly because of the space issue, but fancy new algorithms probably consume more CPU than even the latest mobile hw can manage[1]
The second, related issue is that, again, in order to do this kind of open domain recognition, you need to constantly improve the models[2]. Even a moderately sized set of models would be a pain in the rear to send back up to every android device using the system.
[1] You can typically do the front-end feature processing (and adaptation) on the device, and most vendor's solutions end up doing a bit of that.
[2] Even for speaker dependent modelling (dictation), you'd probably want to associate a key with the model and adapt that model in the cloud rather than sending model updates back to the device because of the space and transmit times.
Not client-side, or I would use it more often (lot of dead spots near my house due to the peculiar geography).
Update: I just noticed that this is actually an implementation of the HTML Speech Input API (http://lists.w3.org/Archives/Public/public-xg-htmlspeech/201...), so technically this could appear on any site. This makes things actually a lot more interesting :)
http://chrome.blogspot.com/2011/03/talking-to-your-computer-...
It's the first HTML standard that I can think of where this is true.