Ask HN: Why is there no high quality method for voice control of a PC?

Speech models today can mine the entire corpus of published conversation and return the most likely response to a given statement. That's not how we converse. Every relationship you have is a little model in your brain that we call a person's "personality." Every one talks differently, has different frames of reference, uses different codes of language, different assumptions. Cutting edge speech models work perfectly for the perfectly average speaker, but that person does not exist! The farther we stray from the mean, the more alienating these speech models become.

A friend of mine uses VoiceAttack in a few VR games and it seems to work decently for triggering actions. Not sure if it's any good at transcription though.

Google and Siri are good at what they do. They aren't good at other things, such as dictation.

I see the big problem in voice interaction is that a human being will ask you questions to clarify what you said if they don't understand and current systems don't even try. (Actually the search paradigm lets you do some refinement, "Ok Google" works amazingly well on Android TV.)

Superhuman accuracy at dictation doesn't translate to a useful ability to understand text. You're doing great if you only garble 1 out of 20 words. Some errors are inconsequential, but if it garbles every other sentence then you are going to feel 0% understood.

As far as human computer interfaces go, keyboard and mouse probably win comfortably in both bandwidth and latency against speech to text in almost all tasks. Former also requires a less physical effort and is creates less noise for others. My guess is that this shrinks the demand for good quality voice HCI significantly and those who really need it end up being overlooked.

Is there a high quality voice control for anything? I only ever tried Google Assistant and it mostly always can't comprehend queries beyond "timer 10 minutes", like putting a water boiler on schedule.

  "Send a slack to my wife" -> "Sorry, who do you want to text?"

Have you looked at Talon[1] for programming and system control? I used it for a few months last year and while the first two weeks were difficult I was able to nail down a workflow that really suited me. After another few weeks I felt as comfortable and capable working with it as I did a keyboard and mouse. (Cannot attest to its capabilities on Windows)

[1] https://talonvoice.com/

Siri can’t understand “set a timer” more often than 3 in 4 tries for me, and any sentence with more than four words will have one error in it no matter what. I envy you the accuracy your voice assistants offer you, but for me, voice control makes me want to snap my phone in half from frustration at how terrible it is. I still can’t remember why I have a reminder set with the name “2910”, which is the transcription of my spoken English sentence at the time. So at the very least, I imagine the holdup is that voice control failure conditions are miserably bad, when it fails; and, “Delete this sentence” -> “Formatting C:\” misunderstandings are too easy in modern OSes still. (Windows still offers “Format” as a primary context menu choice on the boot hard drive!)

Absent regulation or other incentives to nudge the market otherwise, the overwhelming majority of software is and will be written to use existing input methods--i.e. adding a new input method isn't the core competency of a team creating the world's best todo list app.

With that precondition, any voice-to-control layer on the desktop is in the tough situation of translating between voice input and a piece of software that was designed without voice input in mind.

Google and Siri, etc., aren't as beholden to the desktop/browser interface paradigm, so they don't have to perform this interface translation.

>The market is huge

Apparently ... it's not

Or, rather, it's not YET "huge"

Sure - half the planet is online, but they're speaking myriad languages in more combinations of enunciation, dialect, and accent than is probably even calculable

>the Natural Language Processing of "OK Google" and Siri are quite refined at this point

Totally different to ask for today's weather and to tell a computer what to do - just like it's totally different to hit your favorite search engine and type "what is Pluto's orbit" and to write the search engine that goes off and does what you asked (and even when it does go off and do it, it still returns multiple (often conflicting) results - which leads to the whole problem of identifying authority online (something I wrote about 15+ years ago https://antipaucity.com/2006/10/23/authority-issues-online/#...))

It's also worlds different to be able to respond to variations on a theme of maybe a couple hundred search keywords (is it even that many?) and the literally unlimited number of commands people issue to their computing devices every day. Let's even say Siri is That Good™ - you've got a MacBook, iPhone, and iPad on your desk ...which one should respond when you say, "Hey, Siri"? Why that one vs this one? Do you have to start every command with the name of the device? Maybe that's not so hard at home (maybe), but get into corporate environments with naming conventions like H5GG71WLD? ... or dozens/scores/hundreds of people within listening distance of everyone's microphones getting triggered by other conversations in the room, conference calls, your cubemates' inability to attenuate their voices and aim only at their laptop when talking ...

It's a nightmare to think about - practically, let alone computationally

Most people look at the example of, say, Star Trek for voice commands to "the computer". Ever notice the computer only responds when the script demands it? Geordi shouting in Engineering commands to his team or panicked messages to the bridge are never misinterpreted by the computer as commands to it

That's mighty convenient - and not at all representative of anything resembling a reality we can create [yet]

Maybe in another few decades or centuries ... but I'd wager probably not

Another consideration: speaking is very slow compared to a click, tap, or typing a few characters at a prompt. Why would you want to intentionally make your human-to-device interactions more clumsy and error-prone?

I've heard cursorless (https://github.com/cursorless-dev/cursorless-talon) is good but have never tried it. Syntax-aware voice navigation of code, powered by tree-sitter queries!

I also have a friend who is a gifted programmer who lost his ability to type about a decade ago; he has put together an open-source software stack to help: http://www.cs.columbia.edu/~dwk/

Of course this doesn't really answer your question. But it's a hard problem, and you're basically forced to become a power user to reliably interact with your PC.

That sounds exhausting.

"Open this program"

"Minimize"

"Focus on this text input"

..dictate..

"switch to command mode"

"save and close"

i'd rather just: "click click tab type ctrl-S"

For anyone interested in this topic, you might be interested in this tech talk[1] from Emily Shea. In it she demos a tech stack similar to what's mentioned here, to fairly good effect. It does appear that required a lot of tweaking on her part and is optimized for the writing of code, and I'm not sure how well it functions for more general contexts.

1: https://www.youtube.com/watch?v=YKuRkGkf5HU

... because it's not a good way to control a computer?

  Mac | Anti-Mac
  Direct Manipulation | Delegation
  See and Point | Describe and Command
  WYSIWYG | Represent Meaning
  User Control | Shared Control
  Feedback and Dialog | System Handles Details
  Forgiveness | Model User Actions

Once automatic speech recognition (ASR) gets closer to bullet-proof, I expect this to become a huge thing, but right now, it seems like you're getting better error rates than typical.

Any input method where you frequently have to repeat yourself and undo things won't get mainstream. I'd bet people's mainstream tolerance for errors would have to be like one per five to ten minutes before you could get them to really adopt something like this (barring disability reasons, like RSI). Until then, the tech and market don't match.

Voice input is good for high level tasks and goals, requiring a high level comprehension.

For detailed work though the more direct method of translating movements is far more efficient.

When you can describe an abstract end goal voice is great. When you have to actually do all the individual steps towards some high level goal then it's like telling a newbie programmer through some high level database optimization. You only use voice because your main goal here is to teach someone. If the PC could be taught that way, then voice would be in demand for such tasks too.

Certainly exists and I have setup this for various blind people who make due. Unfortunately dont recall what it was exactly but they bought it and all that.

The thing about voice is how weak it is. Even if you've well trained it and you speak well, which i don't. It wont be as good as a keyboard.

Putting work into voice like this for productivity is pointless. Any effort is best placed in brain computer interfaces. Hopefully not surgically required, like neurolink is doing. More of a headset like Valve and openbci is doing.

Lets just wear a headset and work, keyboards can just be there in case you need them.

Have you read this blog post by Josh W. Comeau outlining his experience with Talon for a developer workflow?

https://www.joshwcomeau.com/blog/hands-free-coding/

I've kept this bookmarked for when this eventually happens to me. https://arstechnica.com/gaming/2019/04/coding-without-a-keys...

I'm in the same boat as you, OP. Talon has proven a lifesaver ... or at least it promises to be one. I'm still getting used to it.

My finding, for text dictation (not code), is that even halfway decent dictation, such as is available on iPhone, still needs much post-dictation editing. I feel that the biggest impact to be made in this area is superior capabilities for this editing phase.

I summarized and wrote up my thoughts as a grant proposal for Scott Alexander's recent "micro grants" project. Get in touch (email in my profile) if you want to read that, or if you'd like to talk about dictation, voice control, voice coding, and editing operations -- or just get some moral support.

I’ve been messing around with https://github.com/ideasman42/nerd-dictation which, with the big model, gives surprisingly accurate local detections. Definitely more diy/hacker focused than actually being a solution though.

I'll give an answer in a slightly separate direction: there aren't engines that are both good enough and open enough to hook into that Open Source communities can build around them.

There are two ways that new software gets built: either the market is big enough and accessible enough that commercial software gets built, or the software is easy enough to build that hobbyists enter the space and solve their own problems. For example, the commercial market for keyboard-driven interfaces is also quite small, but we still have stuff like Sway. But a good keyboard-driven interface is easier to build than speech recognition.

I've been curious about this area for a while, but my understanding is voice-to-text Open Source solutions are still kind of primitive for general text transcribing. The libraries aren't very fun to work with, they're often embedded Python/Java "stuff", and the accuracy isn't great if you advance past the level of text transcription. Additionally, controlling computers and hooking into X or Wayland feels a bit hacky.

That being said, I'll push back on people who are saying that no one would want to control an interface this way. The success of systems like Alexa/Siri/Google are pretty definitive proof to me that (all their weaknesses side) there is a market for voice interfaces. But the ties between that market and the desktop are not strong, and the ecosystem isn't open enough to really build on in that direction.

I suspect that until efforts like Mozilla's open speech datasets pick up more steam and become competitive (if they ever do), it's going to be kind of laggy to find solutions because it's not immediately obvious how to enter the market, either as a commercial company or as an Open Source dev. But maybe I'm wrong and I just haven't researched it enough and the area is totally ripe for disruption. Maybe for people with RSI they'd tolerate something like clipping a bluetooth mic to their lapel or something and that would boost accuracy. Maybe there's another way to approach entering code that isn't just straight text recognition, possibly combining it with some kind of AST or code analysis that made it easier to guess what people were saying.

In any case, I don't think the problem is that people don't want to talk to their computers. Personally I don't like using voice assistants, but they are very popular, in no small part because of the voice part. So maybe there is an evolution of desktop UI controls that could become really popular, or at least competitive with entrenched solutions for people with limited mobility or RSI. But it would require someone to introduce some kind of actual UX innovation into the space, or to find a way of getting over the moat around good recognition and OS integration.

Because Google is keeping all their crowdsourced voice data secret.

Wow, so many replies that boil down to “because typing is better, you should just type”.

This is fairly insulting as RSI’s are very much a real thing.

Does this community also think that wheelchair ramps should never be invested in because stairs are clearly superior?

I’d rather see the brain power in this community focused on solutions. Keyboard + mouse have lasted so long because they work surprisingly well, but I hope there is a day that we dream up something better that does not require slowly giving ourselves carpel tunnel.