So how can one actually use one of these systems? I'm familiar with pocketsphinx, where you download it and then run procketsphinx_continuous and it prints a transcription of the microphone. Or Julius, where you write a grammar and run it with that and it prints the output. The process for DeepSpeech seems to be along the lines of "get access to a machine with ten GPUs, find a huge dataset, train it, and then run that somehow" - are there compiled models that can actually be shipped as part of an application?
Wasn't aware of Picovoice. Just tried the live do they have on their website... wow, it's... not great! Even if I spoke as precisely as possible or/and put on an American accent, it was way off the mark.
Someone in that thread ported it to linux. This demo is just "acoustic model emissions", which are character level predictions (with no repeated characters), but I have "decoding" (turning into english sentences) working locally as well and I'll post a new demo at some point.
I'm a little confused about the title because the first paper is from 2014.
It's also too bad this doesn't mention any traditional HMM-based ASR techniques, as HMMs continue to be used on many SOTA systems, particularly those that can be reproduced publicly: https://github.com/syhw/wer_are_we
The article quotes DeepSpeech, wavenets, LSTMs of all sorts; essentially, all the neural networks that scale terribly. DeepSpeech for example is pretty heavy and requires a decent GPU to get anywhere near a realtime factor of 1.
Meanwhile ASR through HMM's consistently hits realtime factors sub-1 and can run on small CPUs. e.g. the default models Kaldi ships with outperform DeepSpeech on a lot of modern examples, and are exponentially faster.
Moreover, training these HMM's is something that is feasible for a normal developer. Training newer models requires data of scale and quality (iirc Mozilla's models are trained on Common Speech which is an enormous crowd sourced dataset, and Google's wavenet models use an internal dataset of very high quality and quantity).
Until the models get more practically achievable, ASR for average people will probably continue to be dominated by Kaldi, Sphinx etc
Do you have any advice or links for a "normal" developer to get started with HMMs for speech recognition? I'd love to build something just for myself here.
^^ If you're getting started, just following the steps here can get you set up really fast. IIRC there's also an HTTP endpoint at /recognize you can use instead of WebSocket if you're transcribing audio files so it's pretty cool!
Note we're talking about LVCSR, you're not going to get 0.01 RTF doing that. Still, wav2letter is quite impressive from the numbers I've seen (although you're still going to need an insane amount of compute and data to train a good model). I've been meaning to try it out, but the setup is so complicated I haven't gotten around to it (I tried it and at some point while going down the dependency tree I said "fuck this" and stopped).
<0.01x in this case is fully encoding/decoding arbitrary input speech against a 700k word english wikipedia-based language model, for online/realtime use. I'd say it's pretty large vocabulary, and it's continuous in the sense that you can talk forever and it will output text regularly.
(Wikipedia doesn't make the best LM, I just wanted to test with something that knew about a lot of interesting english words)
Going to be honest but I don't think that is physically possible (i.e. I don't believe that, no offense). Unless you're using a very small beam and model.
1. I don't have the quad core CPU we hit <0.01x on, I only have a dual core laptop CPU. (The quad core numbers were with an i7 7700k or so, they came from the other person I'm working on this with).
2. I'm not running a parallel decode. That's in a branch from my collaborator I haven't merged/built myself yet since I'm only on a dual core.
3. Screen capture has a CPU hit and seems to have slightly increased my RTF during recording.
Haha nice one. Okay yeah I believe it (can imagine 0.02, which is close enough). Although I have to mention judging by when the text appears it's above 0.5, but I'm guessing you only output the text after a pause or something.
Sure, that's probably the VAD, which can be tuned (the VAD decides if your phrase is done or not yet). That's part of why I included activity monitor watching w2l, so you could see it wasn't using any CPU while I talked. Would probably be more convincing if I printed something at the moment the VAD cut off the user so you could visually judge from the boundary of what we considered their speech.
My setup is a work in progress, see the sibling thread. My 2015 2-core MacBook is probably not faster than a t2.medium, so you should be able to hit the same sort of 0.05x ballpark numbers easily with the same sort of setup.
Is there a SIMD.js / WASM equivalent optimized convolution / GEMM? That's pretty much all we'd need to port this to web... well, that and maybe a language model that isn't 1GB. The wav2letter acoustic model I'm using is based on the librispeech conv_glu, which is almost entirely served by conv1d layers.
I've honestly already been considering a demo for my main project (which is mixed english / command decoding) that runs entirely in a web page, if you have engineering time to throw at your christmas wish, we should talk :P
Two years ago someone tweaked Kaldi to build to wasm (https://github.com/adrianbg/kaldi.js). AFAIR it ran at decent speed in a browser (with small models), but it hasn't been maintained since.
Are there any good quality OSS speech recognition libraries that are easy to get started with, or is it still so complex/expensive that this is a fantasy?
I really love the idea of hacking something together for development so I don't need to use my arms and hands so much, or could lessen my mouse use (for accessibility reasons), but I don't know how realistic that is.
Edit: I looked into cloud-based ASR, such as that provided by Azure and AWS, but that would mean network latency on top of the recognition latency, and that would drive me nuts!
I dream of building a competitor to Siri and google and I’d probably use https://snips.ai/. I think it gains recognition accuracy by having a limited skill set. Looks good though and has functionality for defining skills.
Speech recognition of a small set of possible words is basically a solved problem. It's why you've been able to call a phone support line and read numbers to it for years and years now; speech recognition for 10 digits and a handful of control words is basically done.
So, if your project can be built on that, good news; you can build now.
I've been trying out Snips, it's pretty cool and works reasonably well. A lot of the overall system is open source and runs offline, but the training happens on their servers and is closed source AFAIK. You download the trained model and can run it on a raspberry pi etc offline. But they claim that what Snips offers for free isn't nearly as good as what the commercial offering does. My priorities unfortunately shifted away from finding out about the quality difference.
IIRC Snips's business model is to create custom voice agents that run offline for other companies and services around that, eg. custom hotwords etc.
It used the windows version of Dragon Dictate that had a python interface which was then hooked up to emacs, iirc. I attempted to replicate his system at some point but never got it working -- seems that the libraries he referenced weren't well maintained and/or running windows through a VM on my mac introduced additional issues.
(Slightly offtopic, sorry) I think the best path to “programming by conversation” is to have a programming language designed specifically for that goal, so perhaps you want to research that. I’d say a version of prolog / attempto controlled english / similar functional language (to work on small bits of mostly contained meaning), a verbal repl that is carefully designed to explore data, generate test data, manage errors correctly, and highlight what’s important, etc. would be a super interesting project, that can then be easily plugged into voice interfaces.
Doesn't seem particularly OT to me - it's an interesting idea!
I'm not totally convinved that the language needs to be designed specifically for voice coding though - I can see how having an IDE designed for it would be a huge bonus though... damn, another interesting side project to add to the list!
38 comments
[ 5.3 ms ] story [ 99.2 ms ] threadFacebook has open sourced some pre-trained models: https://github.com/facebookresearch/wav2letter
Picovoice has some smaller, more efficient models capable of running on edge devices: https://github.com/Picovoice
Full ASR does require quite large models and datasets, but you don't need nearly that much power or data to fine-tune a model for your own domain.
https://github.com/facebookresearch/wav2letter/issues/327
Someone in that thread ported it to linux. This demo is just "acoustic model emissions", which are character level predictions (with no repeated characters), but I have "decoding" (turning into english sentences) working locally as well and I'll post a new demo at some point.
They have an example that accepts streaming from the microphone: https://github.com/mozilla/DeepSpeech/tree/master/examples/m...
See the last full release here: https://github.com/mozilla/DeepSpeech/releases/tag/v0.5.1
It's also too bad this doesn't mention any traditional HMM-based ASR techniques, as HMMs continue to be used on many SOTA systems, particularly those that can be reproduced publicly: https://github.com/syhw/wer_are_we
The article quotes DeepSpeech, wavenets, LSTMs of all sorts; essentially, all the neural networks that scale terribly. DeepSpeech for example is pretty heavy and requires a decent GPU to get anywhere near a realtime factor of 1.
Meanwhile ASR through HMM's consistently hits realtime factors sub-1 and can run on small CPUs. e.g. the default models Kaldi ships with outperform DeepSpeech on a lot of modern examples, and are exponentially faster.
Moreover, training these HMM's is something that is feasible for a normal developer. Training newer models requires data of scale and quality (iirc Mozilla's models are trained on Common Speech which is an enormous crowd sourced dataset, and Google's wavenet models use an internal dataset of very high quality and quantity).
Until the models get more practically achievable, ASR for average people will probably continue to be dominated by Kaldi, Sphinx etc
^^ If you're getting started, just following the steps here can get you set up really fast. IIRC there's also an HTTP endpoint at /recognize you can use instead of WebSocket if you're transcribing audio files so it's pretty cool!
(Wikipedia doesn't make the best LM, I just wanted to test with something that knew about a lot of interesting english words)
1. I don't have the quad core CPU we hit <0.01x on, I only have a dual core laptop CPU. (The quad core numbers were with an i7 7700k or so, they came from the other person I'm working on this with).
2. I'm not running a parallel decode. That's in a branch from my collaborator I haven't merged/built myself yet since I'm only on a dual core.
3. Screen capture has a CPU hit and seems to have slightly increased my RTF during recording.
Here's your comment read with 0.05x-0.10x on a dual core CPU: https://youtu.be/jIgUKwR-LaA
Is that enough to convince you that with a stronger CPU and parallel decode we can hit 0.01x?
This video can be partially reproduced with the code and models from here: https://github.com/facebookresearch/wav2letter/issues/327 but most of our decoder optimizations post-date when I compiled that demo.
Some of the later code is here:
https://github.com/talonvoice/wav2letter/tree/w2lapi_static
https://github.com/ckamm/wav2letter/tree/wip-parallel
Here is the exact timing loop code I used in that demo: https://bochs.info/p/nhz5cv
I definitely think it's a big step in the right direction; it's easily 100x faster than DeepSpeech for us.
If I could have anything I wanted for xmas, I'd ask for a speech to text system that is fast enough to work in browser thru wasm or something.
Is there a SIMD.js / WASM equivalent optimized convolution / GEMM? That's pretty much all we'd need to port this to web... well, that and maybe a language model that isn't 1GB. The wav2letter acoustic model I'm using is based on the librispeech conv_glu, which is almost entirely served by conv1d layers.
I've honestly already been considering a demo for my main project (which is mixed english / command decoding) that runs entirely in a web page, if you have engineering time to throw at your christmas wish, we should talk :P
I really love the idea of hacking something together for development so I don't need to use my arms and hands so much, or could lessen my mouse use (for accessibility reasons), but I don't know how realistic that is.
Edit: I looked into cloud-based ASR, such as that provided by Azure and AWS, but that would mean network latency on top of the recognition latency, and that would drive me nuts!
So, if your project can be built on that, good news; you can build now.
But I'm unsure of what Snips actually is - I had a look at the website, but I don't know if this is OSS, commercial software, a library or what?
https://www.youtube.com/watch?v=OWyMA_bT7UI
It used the windows version of Dragon Dictate that had a python interface which was then hooked up to emacs, iirc. I attempted to replicate his system at some point but never got it working -- seems that the libraries he referenced weren't well maintained and/or running windows through a VM on my mac introduced additional issues.
I'm not totally convinved that the language needs to be designed specifically for voice coding though - I can see how having an IDE designed for it would be a huge bonus though... damn, another interesting side project to add to the list!