Ask HN: Non-cloud voice recognition for home use?
I'd like a kind of echo dot like thing running on a set of raspberry pi devices each with a microphone and speaker. Ideally they'd be all over the house. I'm happy if they talk back via wifi to a server in my office for whatever real processing. The server might have 16 cores and 128Gb ram. Might even have two of these if required.
What options do I have? What limits? I'd really prefer answers from people who have experiences with the various options.
If it helps I'm happy to reduce vocabulary to a dictionary of words as long as I can add more words as necessary. Training is also ok. I've already analysed my voice conversations with an echo dot and the vocabulary isn't that large.
Please remember: home use, no off-site clouds. I'm not interested in options involving even a free voice speech-to-text cloud. This eliminates google voice recognition, amazon etc. They are great but out of scope.
So far I've identified CMU Sphinx as a candidate but I'm sure there are others.
Ideas?
129 comments
[ 4.8 ms ] story [ 197 ms ] thread[1] https://mycroft.ai/
Alternatively, you could just fork the Almond project directly and take it from there: https://github.com/stanford-oval/almond-cloud
Thanks.
[1] https://news.ycombinator.com/item?id=17532003
[2] https://news.ycombinator.com/item?id=17534793
[3] https://github.com/stanford-oval/almond-cloud/blob/master/do...
However, the words "significantly more challenging to manage" are straight from their documentation that I linked which I think makes it worth pointing out.
Whether or not it is too challenging is for each individual to decide for themselves.
"You must also deploy custom NLP models, as the official ones will not be compatible. Use this setup only if you absolutely need custom Thingpedia interfaces, and cannot provide these interfaces on Thingpedia."
Like Pebble.
Though there are about 100 repositories there. I am not sure if it is easy to put it all together.
https://makezine.com/2020/03/17/private-by-design-free-and-p...
https://github.com/DragonComputer/Dragonfire
There's a minimal demo app I put together here too: https://github.com/nmstoker/SimpleSpeechLoop
https://github.com/jaxcore/deepspeech-plugin
AFAIK DeepSpeech works by using the neural net to detect characters from speech, and then the language model is used to try to make a sentence out of the character stream, by doing a kind of graph search. Thus if the language model doesn't contain the words you want it to recognize, it'll have a hard time giving good output.
Anyway, I used the following tutorial[1] as a base to build the language model. For the kenlm tools I used Ubuntu WSL, and the generate_trie executable was part of the DeepSpeech native tools package for Windows.
[1]: https://discourse.mozilla.org/t/tutorial-how-i-trained-a-spe...
For my case I just wanted to train it on like 30 different sentences, that took less than a second. But for a general assistant ala Google Home you'll want a large number of sentences and I hear it can take a while (hour or few?).
Due to using probabilities it will match words in other sentences than what you give it, but from my understanding it will be partial to the ones you feed it if DeepSpeech mis-classifies a character or two.
https://youtu.be/LWUBK6PAaxM
This was on a slightly earlier version, and they've made improvements in speed and quality of recognition since then.
I should clarify this. As I mentioned, training the neural net part requires tons of audio and the corresponding text (and people should totally contribute[1], the resulting data sets are released to the public). The neural net in DeepSpeech is then used on an audio stream and outputs a stream of characters.
Turning that stream of characters into sentences is what the language model is for.
Training the neural net is very data and compute intensive, but fortunately Mozilla provides pre-trained models.
Generating the language model is relatively cheap. And if your target language shares sounds with English, you may get away with using the English-trained neural net but with a non-English language model.
[1]: https://voice.mozilla.org/
https://www.researchgate.net/publication/336532830_German_En...
The results are, that adopting works good, but as of writing of the paper somemonths ago, the results were not very good yet. So it takes better trained models for other languages. English seems to be quite good.
What surprised me, is that it works offline very fast even on a rasperry pi!
https://www.hackster.io/dmitrywat/offline-speech-recognition...
Sphinx is just for the automatic speech recognition (ASR) part. But there are better solutions for that:
Kaldi (https://kaldi-asr.org/) is probably the most comprehensive ASR solution, which yields very competitive state-of-the-art results.
RASR (https://www-i6.informatik.rwth-aachen.de/rwth-asr/) is for non-commercial use only but otherwise similar as Kaldi.
If you want to use a simpler ASR system, nowadays end-to-end models perform quite well. There are quite a huge number of projects which support these:
RETURNN (https://github.com/rwth-i6/returnn) is non-commercial TF-based. (Disclaimer: I'm one of the main authors.)
Lingvo (https://github.com/tensorflow/lingvo), from Google, TF-based.
ESPnet (https://github.com/espnet/espnet), PyTorch/Chainer.
...
I really just want the speech-to-text. Ideally I'd also like it to recognise who's talking. But that's a bonus.
https://github.com/gooofy/zamia-speech/ has been training good [acoustic] models which are worth looking at (including training with robustness against noise). They've also got lots of code and docker images and documentation.
pocketsphinx isn't actually that bad to use with their latest acoustic models and small vocabularies (so its utility depends on your exact use case). But it's not generally good with far field mics/dsp processed audio, not really good with noise, and in my experiments quite not as fast as Kaldi.
Better/larger language models in my experience make a world of difference (esp in the general vocab case) for improving accuracy for either of kaldi or pocketsphinx. Nobody really seems to talk about this(?), since everyone always uses the news corpus from like the 80s as the default language model.
I haven't really ever gotten the various ~deepspeech systems working, so I can't speak to them.
It’s pretty hard to blame lay people when speech reco products like Dragon are widely marketed as “voice recognition”.
If you want to be clear and not just pedantic just call it speaker recognition.
with install scripts: https://github.com/NVIDIA/OpenSeq2Seq/blob/master/scripts/je...
The cloud based solutions are so vastly superior to the current non-cloud solutions that unless you're something of an expert in ASR you're just going to get frustrated. If you're worried about privacy, Google lets you pay a little extra to immediately delete the audio after you send it to their servers.
Cloud = giving out your data
But seriously, it feels sometimes, we are very small minority.
I think there's a group of highly technical people who feel increasingly left behind by 'convenience tech' because of what they have to give up in order to use it
Loss of internet access is not an excuse for ignoring basic voice commands in my opinion.
Privacy is also an important factor but not the primary driver for us.
-----------------
Windows 10 IoT for Raspberry Pi comes with offline speech recognition API.
It was not hard to slap some code together that turns on a light when someone says "banana" at a hackathon.
Sounds like exactly what you need.
>If it helps I'm happy to reduce vocabulary to a dictionary of words
You will do it with an XML grammar file for offline recognition[4].
[1]https://docs.microsoft.com/en-us/windows/iot-core/tutorials/...
[2]https://docs.microsoft.com/en-us/windows/iot-core/extend-you...
Someone's demo project:
[3]https://www.hackster.io/krvarma/rpivoice-051857
[4]https://docs.microsoft.com/en-us/windows/uwp/design/input/sp...
[0] https://github.com/spc-ofp/ObserverLengthSampler
- The setup guide shows a Windows system making a Windows iot version. Can't I just download an iso and flash it to an sdcard with dd? Does it need a license?
- The demo projects show C# and while I can develop in monodevelop, I don't have a Windows to compile it with. Is a C# compiler included in Windows iot's .NET distribution or are there also cross-platform (interpreted) languages that run on Windows iot (e.g. Python3)?
Win10 IoT is written to the SD card, and left to first-boot inside the Pi (this bit takes AGES). While it's doing that you install the Win IoT dashboard toolkit onto your PC (Windows only). The Dashboard will find your Win10 IoT Pi on the network, there's a few demo apps pre installed you can play with. It's a free OS, but you need to pay for a dev licence if you want it to not reboot every 24 hours. (There's also an on screen non-production warning).
Now you fire up Visual Studio, which has gained the ability to build C# apps on ARM. You write a small app, including using the visual form designer, and you debug using the PC based Win IoT emulator, or you deploy it directly (using VS) to the the Pi.
Once you are happy with your app, you have to bake it into a new OS image that gets written to the SD card(s) for proper deployment.
Win10 IoT can only run one app in the foreground. It does not have a classic desktop, which should be fine for embedded or kiosk type applications.
Personally, I found it clunky and slow, even on a fast Pi. There's also a fair amount of restrictions applied to your app (think the same restrictions as an Android or iOS app) so if you are used to your C# app having full (read) access to the machine it's running on, you won't get that on Win IoT.
If you want to develop GUI rich apps on the Pi, there are far better alternatives (Mono, Python/GTK etc. on Raspbian).
This is just my take on Win10 IoT. I'm a Windows guy by profession so I don't have an anti-MS bias here.
I'd be hoping I can also load in text-to-speech as well either separately or as part of same application. From what I've read the windows approach to the Pi is more like an appliance. Your application takes over the whole device. This is fine as long as I can load in more functionality to that application.
I need to read more about this.
Thanks for the pointers.
Haven't used it, but seems very nice.
https://youtu.be/ijKTR_GqWwA
If you're looking for something for the command-line, check out https://voice2json.org
Pretty sure Mycroft is capable of that - in theory - you'll need to config it manually. The standard raspberry pi route isn't powerful enough for local.
Check out reespeaker for a raspberry microphone. You'll want one of the more expensive ones for range. Though at like 40 bucks they're not that wildly expensive.
Make sure it's a rasp 4 since wake word is processed locally. And you probably don't need 128gb RAM. No idea what they use but doubt that much.
128GB is the minimum I use for general purpose servers so this machine would be a repurposed machine rather than something specially ordered or built.
Not sure what FAQ you're reading there. There is a whole section of STT engines you can plug into it.
https://mycroft-ai.gitbook.io/docs/using-mycroft-ai/customiz...
Including a local one. It's gonna be a pain in the ass but as I said it's possible
I've been working with Facebook's wav2letter project and the results (speed on CPU, command accuracy) are extremely good in my experience. They also hold the "state of the art" for librispeech (a common benchmark) on wer_are_we [1]. Granted, that's with a 2GB model that doesn't run very well on CPU, but I think most of the fully "state of the art" models are computationally expensive and expected to run on GPU. Wav2letter has other models that are very fast on CPU and still extremely accurate.
You can run their "Streaming ConvNets" model on CPU to transcribe multiple live audio streams in parallel, see their wav2letter@anywhere post for more info [2]
I am getting very good accuracy on the in-progress model I am training for command recognition (3.7% word error rate on librispeech clean, about 8% WER on librispeech other, 20% WER on common voice, 3% WER on "speech commands"). I plan to release it alongside my other models here [5] once I'm done working on it.
There's a simple WER comparison between some of the command engines here [3] Between this and wer_are_we [1] it should give you a general idea of what to expect when talking about Word Error Rate (WER). (Note the wav2letter-talonweb entry in [3] is a rather old model I trained, known to have worse accuracy, it's not even the same NN architecture).
----
As far as constraining the vocabulary, you can try train a kenlm language model for kaldi, deepspeech, and wav2letter by grabbing KenLM and piping normalized (probably lowercase it and remove everything but ascii and quotes) text into lmplz:
And you can turn it into a compressed binary model for wav2letter like this: There are other options, like using a "strict command grammar", but I don't have enough context as to how you want to program this to guide you there.I also have tooling I wrote around wav2letter, such as wav2train [4] which builds wav2letter training and runtime data files for you.
I'm generally happy to talk more and answer any questions.
----
[1] https://github.com/syhw/wer_are_we
[2] https://ai.facebook.com/blog/online-speech-recognition-with-...
[3] https://github.com/daanzu/kaldi-active-grammar/blob/master/d...
[4] https://github.com/talonvoice/wav2train
[5] https://talonvoice.com/research/
We've identified a dictionary of the types of commands and words we use and have a recording of all our amazon and other commands. Training wave files are not an issue.
Have you had any issues with recognising multiple languages?
Thanks!
I haven't personally tested wav2letter with other languages yet. I know zamia-speech trained a german model, and some users have been talking about training for other languages. I've been helping someone who is training several other languages and they've reported great success as well.
If you want to make a new model from scratch in any language, you'll probably want a couple hundred hours of transcribed speech for it, but it doesn't need to be your own speech. Common Voice is a good data source for that.
The issue I see with talon is its currently mac only. That would however still help one of us who lives on wheels (got a 16" macbook IIRC and a mac mini as well). Different set of use-cases so things would be more relaxed.
I see some hints about a linux version however. I've got windows / linux VMs on the server but no other macs. GPUs will be installed soon when I decom some old gaming rigs.
Plenty to think about.
I mostly mentioned wav2letter@anywhere because it could handle a bunch of audio streams centrally, so you can stream from 16 pis to a central box, and it's very accurate.
Even then it can misinterpret so plenty of room for improvement. My quick POC is around 90% accurate for language detection based on the trigger word.
I know virtually nothing about voice recognition, but my spidey sense tells me that it should be possible with the hardware you specify.
A Commodore 64 with a Covox VoiceMaster could recognize voice commands and trigger X-10 switches around a house. (Usually. My setup had about a 70% success rate, but pretty good for the time!) Surely a 16 core, 128GB RAM machine should be able to do far more.
Here is the Google Chrome Web Speech API demo page: https://www.google.com/intl/en/chrome/demos/speech.html
Google Chrome does use a cloud service.
Edit: Firefox does not support the Web Speech API at this time. There are not currently any offline versions of this API as far as I can tell.
https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_...
EDIT: According to other documentation, it is behind a config flag. If you enable it, it will send the data to Google's API through a Mozilla-operated proxy: https://wiki.mozilla.org/index.php?title=Web_Speech_API_-_Sp...
It looks like there is no offline version :(
We license this on commercial bases but would be open to indy-developer friendly licensing. We offer a trial SDK that makes testing/evaluation super easy (it works for 15min at the time).
Ogi
ogi@keeenresearch.com
Does this mean you have a standing offer to train a new language on demand?
Now you know why I want "off-site cloud" for particular things. Not everything. Just what matters. A few things here and there.
There's nothing wrong with the cloud and nothing stops me from using its benefits. Except one minor detail. My use-cases are likely different from others. I wish to keep cloud access as needed but also just a little bit more.
What actions are you looking to handle with the assistant?
Reason I ask is because a voice assistant is a command line interface with no auto-complete or visual feedback. It doesn’t scale well as you add more devices or commands to your home, because it becomes impossible to remember all the phrases you programmed. We’ve found the person who sets up the voice assistant will use it for simple tasks like “turn off all lights” but nobody else benefits and it gets little use beyond timers and music. They are certainly nice to have, but they don’t significantly improve the smart home experience.
If you’re looking to control individual devices, I suggest taking a look at actual occupancy sensors like Hiome (https://hiome.com), which can let you automate your home with zero interaction so it just works for everyone without learning anything (like in a sci-fi movie). Even if you’re the only user, it’s much nicer to never think about your devices again.
Happy to answer any questions about Hiome or what we’ve learned helping people with smart homes in general! -> neil@hiome.com
Corporate promotions really ruin the atmosphere... Take your corporate shilling elsewhere. I'm sure OP doesn't appreciate it.
> Please don't post insinuations about astroturfing, shilling, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email us and we'll look at the data.
Besides, the commenter was upfront about their affiliation and offered insights that appear helpful to the discussion.
From elsewhere in the site guidelines:
> Assume good faith
I do mean it that I'm happy to talk about anything smart home related, so feel free to email me! I spend a lot of my time helping people plan out their smart home build from scratch, including choosing between types of light switches, thermostats, platform, etc unrelated to our sensors.
I did ask for options after all.
The only gotcha is the requirements will overwhelm most basic setups fairly quickly. Not talking 20 lights here or a tv or desk or two.
I've replied to your original comment, thanks.
PS I forgot to mention, all this exists right now as a mixture of amazon echo and a set of text inputs via phone over wifi for evaluation. As we implement more it turns into more voice control and less of a giant hack. So some bits are half-baked but functional.
Around 100 people need separate profiles, each should be able to set alarms, timers, reminders, etc. if they want a routine to create any of those or tell them time or date or temperature they should be able to do that from any of the voice assistants in any room. They might only want such a routine in a particular room. They should be able to define a home device and a current device. Home device would usually be a bedroom for those of us that need them etc.
I definitely don't want to have to create any of those routines etc for any of them. Nothing about these should be fixed in stone. They have to be able to self-serve. We can assume they can navigate the ios amazon app as a baseline level of knowledge.
Room settings include temperature, lighting, curtains, tv on/off, channel, volume to name a few. The voice assistant in some rooms should be able to show web pages on-screen, or even youtube etc. including the laptop someone plugged in on HDMI1.
...the coffee machine automation is also a requirement. Its controlled by a flask app. The voice control should be able to let you order a coffee, strong, black. Or a Dave#5.
We'd also like device detection to trigger when people's phones appear in certain locations.
What kinds of options exist for this?
Are the 100 people using all of the different rooms, or do people mostly stick to their own rooms (like a hotel/dorm)?
I'd love to see what you've built so far, and better understand the problems you're trying to fix. For example, does each room have its own coffee machine, or is it a communal coffee machine? Are the people living here permanently or rotating regularly? What is the goal for device detection (e.g., do you want to use that for presence detection or as a security system or something else)?
We have a prototype for a machine learning system that learns how you use your devices and then automates them by itself, so you don't have to set anything up. Our focus is lights because that's what most people have, but it can also control other on/off things like curtains or tv right now. It sounds like it could be good fit for a situation like this, and I'd be happy to chat with you more on whether it makes sense to try out!
Can you send me an email (neil@hiome.com)?
Would you share the sensor(s) you're using in the device?