Show HN: Willow – Open-source privacy-focused voice assistant hardware (github.com)
I love Home Assistant and I've always thought the ESP BOX[0] hardware is cool. I finally got around to starting a project to use the ESP BOX hardware with Home Assistant and other platforms. Why?
- It's actually "Alexa/Echo competitive". Wake word detection, voice activity detection, echo cancellation, automatic gain control, and high quality audio for $50 means with Willow and the support of Home Assistant there are no compromises on looks, quality, accuracy, speed, and cost.
- It's cheap. With a touch LCD display, dual microphones, speaker, enclosure, buttons, etc it can be bought today for $50 all-in.
- It's ready to go. Take it out of the box, flash with Willow, put it somewhere.
- It's not creepy. Voice is either sent to a self-hosted inference server or commands are recognized locally on the ESP BOX.
- It doesn't hassle or try to sell you. If I hear "Did you know?" one more time from Alexa I think I'm going to lose it.
- It's open source.
- It's capable. This is the first "release" of Willow and I don't think we've even begun scratching the surface of what the hardware and software components are capable of.
- It can integrate with anything. Simple on the wire format - speech output text is sent via HTTP POST to whatever URI you configure. Send it anywhere, and do anything!
- It still does cool maker stuff. With 16 GPIOs exposed on the back of the enclosure there are all kinds of interesting possibilities.
This is the first (and VERY early) release but we're really interested to hear what HN thinks!
140 comments
[ 3.2 ms ] story [ 209 ms ] threadLooks really promising!
If I'm being perfectly honest I'm surprised we got it this far already. If I wanted to be really critical:
- Far-field speech is actually kind of hard. There are at least dozens of "knobs" we can tweak between the various component libraries, etc to improve speech quality and reliability for more users in more environments. We've tested as much as we can considering there's only two of us but we need more testing from more speakers in more environments.
- On the wire/protocol stuff. We're doing pretty rudimentary "open new connection, stream voice, POST somewhere". This adds extra latency and CPU usage because of repeated TLS handshakes, etc. We have plans to use Websockets and what-not to cut down on this.
- We don't really support audio playback yet. For a real "Amazon Echo" type experience you need to be able to ask it random things like "Hey what's the weather outside?" and it needs to "tell" you.
- Ecosystem support. Using the example above, something like Home Assistant or similar needs to know where you are, get the weather, do text to speech, etc for Willow to be able to play it back.
- Other integrations. Alexa has "skills" and stuff and we need to be able to talk to more things.
- UI/UX work. We support the touch display but we did just enough to show colors, print status, add a button, and make a touch cursor that follows your finger around. We also only give audio feedback with a kind-of annoying tone that beeps once for success and twice for failure.
- Speaking of failure, we don't do a great job of telling you what went wrong and where.
- Configuration and flashing. It's very static and has multiple steps. There are all kinds of things that need to get done to make Willow easy enough for less-technical users to deploy and actually use daily without any hassle.
- Local command recognition. It's very early but as noted in the README, wiki, etc the ESP BOX itself can recognize up to 400 commands directly on the device. In testing it works surprisingly well but we have a lot of work to do to make it actually practical for most people.
- Open sourcing our inference server. We plan to do this next week!
Faster than Alexa (and only going to get faster)[0].
Between the far-field speech optimizations provided by the ESP BOX and Espressif frameworks and our inference server (open sourcing next week) using Whisper, and our unique streaming format we've found it to be comparable in terms of quality to Alexa/Echo even with background noise and at distances of up to 30 feet.
[0] - https://www.youtube.com/watch?v=8ETQaLfoImc
Not only are we working on improving performance with the inference server, local on device command recognition is extremely fast. Like "did that really just happen?" fast.
In my local setup when using locally-controlled Wemo switches I swear the latency with local devices is around 300ms or so.
I should make another demo video with that...
That's really cool! Does this mean 400 specific commands, e.g. "turn on the living room lights" or 400 commands that can be applied to different targets, e.g. "turn on the X lights" where X is some light. (400 actually feels like it would be enough to speed up the vast majority of interactions either way, but I'm curious :)
With Home Assistant this means turning on and off two hundred entities. We currently pull light and switch entities from Home Assistant and build the local Multinet speech grammar.
We have goals for better dynamic and adaptive configuration of Willow and part of that is using a Willow Home Assistant component with user configuration inthe HA dashboard, etc to easily select entities, define commands, etc and dynamically update all associated Willow devices.
We feel that with this 400 commands is enough to be practical and useful. Additionally, because the Multinet model returns probability on match to command "fuzzy matching" actually works quite well where "light", "lights", and slightly mis-worded commands still match correctly.
I'm curious if this is something lightweight enough that might be possible to run as a Home Assistant add-on on relatively low-powered hardware such as an RPi.
We'll also (likely) be creating our own TTS/STT HA component for our inference server that does some special/unique things to support Willow.
[0] - https://github.com/toverainc/willow/wiki/Home-Assistant
> - On the wire/protocol stuff. We're doing pretty rudimentary "open new connection, stream voice, POST somewhere". This adds extra latency and CPU usage because of repeated TLS handshakes, etc. We have plans to use Websockets and what-not to cut down on this.
I've recently used the Noise protocol[1] to do some encrypted communication between two services I control but separated by the internet.
It was surprisingly easy!
[1]: https://noiseprotocol.org/
I talk about websockets because they achieve our mission and goal (in this case shaving milliseconds off command -> action -> confirmation) with robust, battle-tested client implementations already available in the ESP framework libraries. Same thing for MQTT. Both are supported by Home Assistant (and almost everything else in the space) today.
Because of this existing framework support, we'll have websockets done today-ish. Then we can (for now) move on to all of the other things people have asked for :). Hah, priorities!
Not saying Noise won't/can't ever happen - just that this is a very ambitious project as it stands and we have plenty of work to do all over the place :)!
Want to write a noise implementation for ESP IDF :)?
A former Mycroft dev, Michael Hansen[1], is still building several year-of-the-voice projects after he was let go. I'm especially excited about Piper[2], which is a C++/py alternative to Mimic3.
[1] https://github.com/synesthesiam [2] https://github.com/rhasspy/piper
Our inference server (open source, releasing next week) has highly optimized Whisper, LLaMA/Vicuna/etc, text to speech, etc implementations as well.
It's actually not that hard on the device - if the response from the HA component has audio, play it.
We just don't have the HA component yet :).
None.
The ESP BOX and ESP SR speech recognition library from Espressif handles the low-level audio stuff like wake word detection, DSP work for quality voice, voice activity detection, etc to get usable far-field audio. The wake word engine uses models from Espressif with wake words like "Alexa", "Hi ESP", "Hi Lexin", etc. If we get traction Espressif can make us a wake engine model for whatever we want (we're thinking "Hi Willow") but open to better ideas!
We currently stream audio after wake in realtime to our very high performance (optimized for "realtime" speech) Whisper inference server implementation. We plan to open source this next week.
We also patched in support for the most recent ESP SR version that has their actually amazingly good Multinet 6 speech command model that does recognition of up to 400 commands completely on device after wake activation. We currently try to pull light and switch entities from your configured Home Assistant instance to build the speech commands but it's really janky. We're working on this.
The default currently is to use our best-effort hosted inference server implementation but like I say in the README, etc we're open sourcing that next week so anyone can stand it up and do all of this completely locally/inside your walls.
For speech to text there are two options and more to come:
1) Completely on device command recognition using the ESP SR Multinet 6 model. Willow will (currently) pull your light and switch entities from Home Assistant and generate the grammar and command definition required by Multinet. We want to develop a Willow Home Assistant component that will provide tighter Willow integration with HA and allow users to do this point and click with dynamic updates for new/changed entities, different kinds of entities, etc all in the HA dashboard/config.
The only "issue" with Multinet is that it only supports 400 defined commands. You're not going to get something like "What's the weather like in $CITY?" out of it.
For that we have:
2-?) Our own highly optimized inference server using Whisper, LLamA/Vicuna, and Speecht5 from transformers (more to come soon). We're open sourcing it next week. Willow streams audio after wake in realtime, gets the STT output, and sends it wherever you want. With the Willow Home Assistant component (doesn't exist yet) it will sit in between our inference server implementation doing STT/TTS or any other STT/TTS implementation supported by Home Assistant and handle all of this for you.
[0] - https://github.com/espressif/esp-sr
Our inference server also supports long-lived sessions via WebRTC for transcription, etc applications ;).
You can chain speech to text -> LLM -> text to speech completely in the inference server and input/output through Willow, along with other APIs or whatever you want.
For wake word and voice activity detection, audio processing, etc we use the ESP SR (speech recognition) framework from Espressif[0]. For speech to text there are two options and more to come:
1) Completely on device command recognition using the ESP SR Multinet 6 model. Willow will (currently) pull your light and switch entities from Home Assistant and generate the grammar and command definition required by Multinet. We want to develop a Willow Home Assistant component that will provide tighter Willow integration with HA and allow users to do this point and click with dynamic updates for new/changed entities, different kinds of entities, etc all in the HA dashboard/config.
The only "issue" with Multinet is that it only supports 400 defined commands. You're not going to get something like "What's the weather like in $CITY?" out of it.
For that we have:
2-?) Our own highly optimized inference server using Whisper, LLamA/Vicuna, and Speecht5 from transformers (more to come soon). We're open sourcing it next week. Willow streams audio after wake in realtime, gets the STT output, and sends it wherever you want. With the Willow Home Assistant component (doesn't exist yet) it will sit in between our inference server implementation doing STT/TTS or any other STT/TTS implementation supported by Home Assistant and handle all of this for you - including chaining together other HA components, APIs, etc.
[0] - https://github.com/espressif/esp-sr
Other than low level issues in the Espressif wifi stack (which is very robust, mature, and has been beat on heavily) I don't see any potential security issues.
That said the old expression "it's easy for someone to design a lock they can't pick" certainly applies.
We'd welcome someone owning it and bringing any issues to our attention!
How does the esp-box compare? E.g. in a noisy environemnt, tv in the background, kids and dogs running around?
Between that and the Espressif AFE (audio frontend interface) doing a bunch of DSP "stuff" in our testing it does remarkably well in noisy environments and far-field (25-30 feet) use cases.
Our inference server implementation (open source, releasing next week) uses a highly performance optimized Whisper which does famously well with less-than-ideal speech quality.
All in, even though it's all very early, it's very competitive with Echo, etc.
With local command recognition with Willow I can turn my wemo switches on and off, completely on device, in roughly 300ms. That's not a typo. I'm going to make another demo video showing that.
We also support live streaming of audio after wake to our highly optimized Whisper inference server implementation (open source, releasing next week). That's what our current demo video uses[0]. It's really more intended for pro/commercial applications as it supports CPU but really flies with CUDA - where even on a GTX 1060 3GB you can do 3-5 seconds of speech in ~500ms or so.
We also plan to have a Willow Home Assistant component to support Willow "stuff" while enabling use of any of the STT/TTS modules in Home Assistant (including another component for our inference server you can self-host that does special Willow stuff).
[0] - https://www.youtube.com/watch?v=8ETQaLfoImc
You'll be able to see when we release our inference server implementation next week that it's more than a version of "realtime" enough to fool nearly anyone, especially with an application like this where you aren't looking for model output in real time. You're streaming speech, buffering on the server, waiting for the end of voice activity detection, running Whisper, taking the transcription, and doing something with it. Other than a cool demo I'm not really sure what streaming ASR output provides but that's probably lack of imagination on my part :).
That said, these are great pointers and we're certainly not opposed to it! At the end of the day Willow does the "work on the ground" of detecting wake word, getting clean audio, and streaming the audio. Where it goes and what happens then is up to you! There's no reason at all we couldn't support streaming ASR output.
I think controlling the odd device, setting a timer, adding items to a shopping list covers about 90% of my Alexa use. The remaining bits are asking it to play music, or dropping into another room. Seems like a good portion of these could be covered already.
My suspicion is Espressif (until now, hah) hasn't sold a lot of ESP Boxes. We were concerned that if Willow takes off they will sell out. That already appears to be happening.
Espressif has tremendous manufacturing capacity and we hope they will scale up ESP BOX production to meet demand now that (with Willow) it exists. The only gaiting item for them is probably the plastic enclosure and they should be able to figure out how to produce that en masse :).
EDIT: Found one with a direct link from Ali from Espressif's site, even though it doesn't show up in a search:
https://www.aliexpress.com/item/1005003980216150.html?spm=a2...
I do not see anything posted on the Home Assistant (HA) Community forums.
> Configuring and building Willow for the ESP BOX is a multi-step process. We're working on improving that but for now...
This is crucial as your "competitors" are ready out of the box. I believe HA can be a Google/Alexa alternative to the masses only if the "out-of-the-box" experience is comparable to the commercial solutions.
Good luck, and keep us updated!
HN was my first stop (of course) - I'll be heading over there shortly to post.
Oh yeah, we're well aware of how much of a "pain" getting Willow going can be. I don't like it (at all).
That said, you configure and build once for your environment and then get a .bin that can be flashed to the ESP BOX with anything that does ESP flashing (like various web interfaces, etc) or you can re-run the flash command across X devices. So even now, in this early stage, it's at least only painful once ;).
Down the road we want to have a Willow Home Assistant component that does everything inside of the HA dashboard so users (like esphome, maybe even using esphome) can point-click-configure-flash entirely from the HA dashboard. Not to mention ongoing dynamic configuration, over the air updates, etc.
I talk about all of this on our wiki[0].
[0] - https://github.com/toverainc/willow/wiki/Home-Assistant
It's actually faster to do it locally. They want it tethered to the cloud for surveillance.
For "basic" command recognition the ESP SR (speech recognition) library supports up to 400 defined speech commands that run completely on the device. For most people this is plenty to control devices around the home, etc. Because it is all local it's extremely fast - as I said in another comment pushing "Did that really just happen?" fast.
However, for cases where someone wants to be able to throw any kind of random speech at it "Hey Willow what is the weather in Sofia, Bulgaria?" that's probably beyond the fundamental capabilities of a device with enclosure, display, mics, etc that sells for $50.
That's why we plan to support any of the STT/TTS modules provided by Home Assistant to run on local Raspberry Pis or wherever they host HA. Additionally, we're open sourcing our extremely fast highly optimized Whisper/LLM/TTS inference server next week so people can self host that wherever they want.
checking the weather in Sofia, Bulgaria requires cloud, current information. it's not "random speech". ESP SR capability issues don't mean that you cannot process it locally.
the comment was on "voice processing" i.e. sending speech to the cloud, not sending a call request to get the weather information.
besides, local intent detection, beyond 400 commands, there are great local STT options, working better than most cloud STTs for "random speech"
https://github.com/alphacep/vosk-api https://picovoice.ai/platform/cheetah/
There are at least two things here:
1) The ability to do speech to text on random speech. I'm going to stick by that description :). If you've ever watched a little kid play with Alexa it's definitely what you would call "random speech" haha!
2) The ability to satisfy the request (intent) of the text output. Up to and including current information via API, etc.
Our soon to be released highly optimized open source inference server uses Whisper and is ridiculously fast and accurate. Based on our testing with nieces and nephews we have "random speech" covered :). Our inference server also supports LLaMA, Vicuna, etc and can chain together STT -> LLM/API/etc -> TTS - with the output simply played over the Willow speaker and/or displayed on the LCD.
Our goal is to make a Willow Home Assistant component that assists with #2. There are plenty of HA integrations and components to do things like get weather in real time, in addition to satisfying user intent recognition. They have an entire platform for it[0]. Additionally, we will make our inference server implementation (that does truly unique things for Willow) available as just another TTS/STT integration option on top of the implementations they already support so you can use whatever you want, or send the audio output after wake to whatever you want like Vosk, Cheetah, etc, etc.
[0] - https://developers.home-assistant.io/docs/intent_index/
We haven't been focused on the ESP BOX Lite because it seems kind of limited. However, Espressif hasn't sold many of these things since release and judging from people looking for stock, etc in this thread I think that's about to change.
Espressif has incredible manufacturing capacity and our hope is they will ramp up manufacture of the ESP BOX family now because (to my knowledge) Willow is the first project that actually makes meaningful use of them.
The only gaiting component of the ESP BOX family that I can see is the plastic enclosure. I'm sure Espressif can figure out how to crank these things out ;).
[0] - https://github.com/toverainc/willow/wiki/Hardware
https://www.espressif.com/en/news/ESP32-S3-BOX_video
Going to try to hack this into something my mom can use (who has trouble with confusion and memory). Could potentially be very great.
Thank you
We are really, truly, and seriously committed to building a device that with support from Home Assistant and other integrations doesn't leave any reason whatsoever to buy an Echo or similar creepy commercial device. No compromises on cost, performance, accuracy, speed, usability, functionality, etc.
We're really looking forward to getting additional testing and feedback from the community on speech recognition results, other integrations, etc. It's just two of us working on this part time over the last month or so - this is VERY early but I think we're off to a good start!
Best of luck with the development! I'll definitely be following closely. Do you sell the pre-built hardware yourself?
When you're releasing a pet project of love like this you never really know if other people are going to appreciate it as much as you do. Looking here on HN it seems like people appreciate it.
We don't sell the hardware currently because:
1) Espressif has well established sales channels and distribution worldwide.
2) It's not our "business model". In my capacity as advisor to a few startups in the space I've been approached by various commercial entities that want a hardware voice interface they fully control. In healthcare, for example, there are all kinds of interesting audio and speech applications but NO ONE, and I mean NO ONE is going to be ok with seeing an Echo in their doctor's office. That's where an ESP BOX or custom manufactured hardware and Willow come in.
Our business model is to combine our soon to be released very high performance inference and API server with Willow to support these commercial applications (and home users with HA, of course). In all but a few identified and very limited cases all work will come back to the open source projects like our inference server and Willow.
https://neon.ai/NeonAIforMycroftMarkII
https://community.mycroft.ai/t/faq-ovos-neon-and-the-future-...
Although MycroftAI, Inc. has ceased development, the Assistant survives.
A few years ago, some of MycroftAI’s partners started using @JarbasAl ‘s code (more information below) which eventually became a fork of the assistant. Now that MycroftAI is unable to continue development, the fork’s devs - The OVOS Foundation - have decided to take up leadership of the Assistant’s development and its open source ecosystem.
MycroftAI has signed over Mark II software support, as well as these forums, to one of those partners, a company called NeonAI. Between the OVOS Foundation and NeonAI, the voice assistant and the smart speaker project are getting a new lease on life.
The OVOS Assistant - it’ll get a better name soon, we promise - started out as a drop-in replacement for Mycroft. It should be compatible with all your classic Mycroft skills. It will even accept existing configuration files! Because we have been operating at a much smaller scale for the past three years, things will seem rough around the edges for a little while. However, we are scaling up. Read on.
The manual install method in the directions is not manual at all. It's a script that calls several python scripts. One has 2660 LOC and installs a root certificate (hard coded in the script itself) because of course, even though you just cloned the whole repo, it still has to download stuff from the internet. According to the code, "This works around the issue with outdated certificate stores in some installations".
Does anyone familiar with espressiff have an actual manual method of installing a dev environment for this device that doesn't involved pwning myself?
For anyone who would try to use this with Willow (I like the effort and CERTAINLY don't love the ESP dev environment as-is):
- ESP ADF is actually the root of the project. ESP-IDF and all other sub-components are components themselves to ADF.
- We use bleeding edge ESP SR[0] that we also include as an ADF component.
- Plus LVGL, ESP-DSP, esp-lcd-touch, and likely others I'm forgetting ATM.
[0] - https://github.com/espressif/esp-sr
Yes, this is why we went through the pain of doing what we're able to do with this hardware.
Even in this initial release it's competitive with Echo, etc even on cost.
But I hate the privacy invasion aspect. I’m definitely in the market for something like this. And this one looks great.
Additionally, I’ve noticed that the google voice assistant (connected to Spotify) doesn’t keep playing the albums I ask for.
It states it’s playing the album. But after 4/5 songs it starts playing different songs, or different artists.
Biggest fundamental issue is the speaker built in the ESP BOX is optimized for speech and not going to impress anyone playing music.
That said, the ESP BOX (of course) supports bluetooth so we can definitely pair with a speaker you bring.
Willow is the first of it's kind that I'm aware of to enable this kind of functionality at anything close to this price point in the open source ecosystem. Either we (or someone else) is likely going to manufacture an improved ESP BOX with market competitive speakers built-in for music playback.
Then it's "just" a matter of actually getting the music audio but we'll figure that out ;).
- Wake
- Get command
- Send to Home Assistant conversation/intents API[0]
- Home Assistant does whatever you define, including what you describe just like it does today
So unless I'm missing something your use case should "just work".
[0] - https://developers.home-assistant.io/docs/intent_index/
But with google I’m stuck with their integration to Spotify. It’s that component I’d like control over, and that’s why I’d use willow.
That and not being spied on in my own home.
Definitely keen for one.
Espressif really did us all a solid with this hardware and their ADF and SR frameworks.
Whether it's cost, being fully assembled and ready to go, and even wake word, AEC, AGC, BSS, NS, etc at least as of now the ESP BOX is essentially impossible to compete with in terms of hardware in the open ecosystem.
I talk about this and more on our wiki pages[0] (check out "Hardware" and "Home Assistant"). In short, the Espressif frameworks we use /technically/ support the "regular" ESP32 but it's so limited (and the ESP BOX/ESP S3 is so cheap) we're not super interested in supporting it.
We're aiming for an end-user experience that's competitive with Echo, Google Home, etc in every possible way - speed, quality, reliability, functionality, and cost.
In fact, we want to crush them on all points to where there's no reason left to buy one of them.
[0] - https://github.com/toverainc/willow/wiki/
So if you spin up somehow an LLM model locally and connect create a HA Assist pipeline with it and than you use Willow(s future release which should be able to leverage the new Assist featre) as a phisical interface than you are golden.
It may be hard or impossible today but I think within months HA and Willow will mature into a state where tha bigges problem will be the training and runing a good enough LLM model locally. But I bet a good amount of hackers are already hard working on that part anyway.
https://community.home-assistant.io/t/using-gpt3-and-shorcut...
I've been trying to adapt it to an offline LLM model, probably a LLaMA-like one using the llm package for Rust, or a ggml-based C implementation like llama.c.
It could even be fine-tuned or trained to perform better and always output only the json.
This could be a good fit with open sourced tovera when that is released.
I like the idea of supporting natural language commands that feel more natural and don't have to follow a specific syntax.
It can also process general LLM requests, possibly using a third-party LLM like Bard for more up to date responses.
Edit: should have read the README more carefully….
We're aware that's controversial and not really applicable to many home users - that's why we want to support any TTS/STT engine on any hardware supported by Home Assistant (or elsewhere) in addition to ESP BOX on device local command recognition.
But for the people such as yourself, and other commercial/power/whatever users our inference server that we're releasing next week that works with Willow provides impressive results - on anything from a GTX 1060 to an H100 (we've tested and optimized for anything in between the two).
We use ctranslate2 (like faster-whisper) and some other optimizations for performance improvements and conservative VRAM usage. We can simultaneously load large-v2, medium, and base on a GTX 1060 3GB and handle requests without issue.
Again, it's controversial but the fact remains a $100 Tesla P4 that idles at 5 watts and has max TDP of 60 watts from eBay with our inference server implementation does the following:
large-v2, beam 5 - 3.8s of speech, inference time 1.1s
medium, beam 1 (suitable for Willow tasks) - 3.8s of speech, inference time 588ms
medium, beam 1 (suitable for Willow tasks), 29.2s of speech, inference time 1.6s
An RTX 4090 with large-v2, beam 5 does 3.8s of speech in 140ms and 29.2s of speech with medium beam 1 (greedy) in 84ms.
Got a Home Assistant Yellow not long ago, so would be nice to get some decent voice control for it.
That seems unusual. I've been using both for the last few weeks while replacing my Homebridge setup, and Siri has been as accurate as Alexa — good enough that I've decided that I can now leave the Alexa ecosystem. To be more specific, both are (conservatively) 95%+ accurate for my home control scenarios.
I currently have a google home and I'm getting increasingly fed up with it. Besides the privacy concerns, it seems like it's getting worse at being an assistant. I'll want my light turned on by saying "light 100" (for light to 100 percent) and it works about 80% of the time, but the others it starts playing a song with a similar name.
I'd be great if this allows limiting / customizing what words and actions you want.
There are at least two ways to deal with this frustrating issue with Willow:
- With local command recognition via ESP SR command recognition runs completely on the device and the accepted command syntax is defined. It essentially does "fuzzy" matching to address your light command ("light 100") but there's no way it's going to send some random match to play music.
- When using the inference server -or- local recognition we send the speech to text output to the Home Assistant conversation/intents[0] API and you can define valid actions/matches there.
[0] - https://developers.home-assistant.io/docs/intent_index/
However, if I spend the time to configure a "scene" with the right presets, Google has no issue figuring it out.
If only it could notice regular patterns about light settings and offer suggestions that I could approve/deny.
If you don't mind my asking, what do you mean "if it hears something interesting"? Is that based on wake word, or always listen/process?
A long while ago, I wrote a little tutorial[0] on quantizing a speech commands network to the Raspberry. I used that to control lights directly and also for wake word detection.
More recently, I found that I can just use more classic VAD because my uses typically don't suffer if I turn on/off the microphone. My main goal is to not get out the mobile phone for information. That reduces the processing when I turn on the radio...
Not high-end as your solution, but nice enough for my purposes.
[0]. https://devblog.pytorchlightning.ai/applying-quantization-to...
Home Assistant is building a voice assistant as part of our Year of the Voice theme. https://www.home-assistant.io/blog/2023/04/27/year-of-the-vo...
As part of our recent chapter 2 milestone, we introduced new Assist Pipelines. This allows users to configure multiple voice assistants. Your project is using the old "conversation" API. Instead it should use our new assist pipelines API. Docs: https://developers.home-assistant.io/docs/voice/pipelines/
You can even off-load the STT and TTS fully to Home Assistant and only focus on wake words.
You will see a lot higher adoption rate if users can just buy the ESP BOX and install the software on it without installing/compiling stuff. That's exactly why we created ESP Web Tools. It offers projects to offer browser-based installation directly from their website. https://esphome.github.io/esp-web-tools/
If you're going the ESP Web Tools route (and you should!), we've also created Improv Wi-Fi, a small protocol to configure Wi-Fi on the ESP device. This will allow ESP Web Tools to offer an onboarding wizard in the browser once the software has been installed. More info at https://www.improv-wifi.com/
Good luck!
First of all, everyone involved in this project has been big fans and users of HA for many years (in my case at least a decade). THANK YOU! For now Willow wouldn't do anything other than light up a display and sit there without Home Assistant.
We will support the pipelines API and make it a configuration option (eventually default). HA has very rapid release cycles and as you note this is very new. At least for the time being we like the option of people being able to point Willow at older installs and have it "do something" today without requiring an HA upgrade that may or may not include breaking changes - hence the conversation API.
One of our devs is a contributor for esphome and we're heading somewhere in that direction, and he's a big fan of improv :).
We have plans for a Willow HA component and we'd love to run some ideas past the team. Conceptually, in my mind, we'll get to:
- Flashing and initial configuration from HA like esphome (possibly using esphome, but the Espressif ADF/SR/LCD/etc frameworks appear to be quite a ways out for esphome).
- Configuration for all Willow parameters from wifi to local speech commands in the HA dashboard, with dynamic and automatic updates for everything including local speech commands.
- OTA update support.
- TTS and STT components for our inference server implementation. These will (essentially) be very thin proxies for Willow but also enable use of TTS and STT functionality throughout HA.
- Various latency improvements. As the somewhat hasty and lame demo video illustrates[0] we're already "faster" than Alexa while maintaining Alexa competitive wake word, voice activity detection, noise suppression, far-field speech quality, accuracy, etc. With local command recognition on the Willow device and my HA install using Wemo switches (completely local) it's almost "you can't really believe it" fast and accurate.
I should be absolutely clear on something for all - our goal is to be the best hardware voice interface in the world (open source or otherwise) that happens to work very well with Home Assistant. Our goal is not to be a Home Assistant Voice Assistant. I hope that distinction makes at least a little sense.
You and the team are doing incredible work on that goal and while there is certainly some overlap we intend to maintain broad usability and compatibility with just about any platform (home automation, open source, closed source, commercial, whatever) someone may want to use Willow with.
In fact, our "monetization strategy" (to the extent we have one) is based on the various commercial opportunities I've been approached with over the years. Turns out no one wants to see an Amazon Echo in a doctor's office but healthcare is excited about voice (as one example) :).
Essentially, Home Assistant support in Willow will be one of the many integration modules we support, with Willow using as many bog-standard common denominator compliant protocols and transports that don't compromise our goals, while maintaining broad compatibility with just about any integration someone wants to use with Willow.
This is the very early initial release of Willow. We're happy for "end-users" to use it but we don't see the one-time configuration and build step being a huge blocker for our current target user - more technical early adopters who can stand a little pain ;).
[0] - https://www.youtube.com/watch?v=8ETQaLfoImc
Flashed this on ESP I had laying around and did NOT have to upgrade HA (which would have made me not try the project).
So question is - what do you think :)?
- ChangeOver to the new bluetooth subsystem broke many integrations. My Bluetooth TRVs still don't work right (again).
- ONVIF support recently broke for an (admittedly shitty old) IP-Webcam. PTZ never worked/was_exposed.
- My USB-connected android devices can't be be controlled by the ADB-Integration anymore. There was some integration renaming/rescoping recently.
Home-Assistant still (imho) is best solution in this space for most combinations of metrics. I'd still recommend it to anyone.
(I tinker a lot with my HA-install/network, so maybe some of the above are issues on my end)
I hate using it. Yet, I have no viable OSS alternatives.