Hate people typing on noisy mechanical keyboards? You're going to love everyone babbling into their devices all the time.
I really don't want to talk to my computer/phone/coffee pot/house. I would rather that software have its normal interface polished and made closer to something that is usable, than have another UI paradigm bolted on that is going to be badly implemented 80% of the time by application developers.
To add on to that most of the voice enabled apps have trouble even recognizing speech when it comes to different dialects of English. For example, Southern dialect, "I am fixing to go to the bar, are you ready" would translate to "I am fixing the bar are you there yet?".
Plenty of technology exists for adaptation to speaker and/or dialect, it's just not considered particularly sexy. To phrase it in economic terms, you can spend your time doing algorithmic work to improve recognition slightly for your existing English users, or you could just deploy a new system for the 70 million or so affluent, tech-literate South Koreans or the maybe a half a billion or so upwardly mobile Hindi/Urdu speakers.
But, give it a little time and you can expect that authenticated users will be getting speaker adaptation.
"We were promised Jarvis, but all we got were eight different closed-source, non-extensible, hosted voice recognition platforms."
What are the leading voice recognition assistant thingies? Siri, Google Now (no human name, much like most google employees), Cortana, Alexis? They each have huge acoustic models and run completely privately and are completely closed source.
We can't have Jarvis until we run our own hackable computing infrastructures personally, not globally hosted versions running behind the walls of Trust Us, Inc.
Which mobile platform will be the first to offer pluggable voice-to-action backends? Much like we pick web browsers, we should be able to select our voice command interfaces, but that would require a lot more interop between hardware manufactures and would also kill their moats of unlimited training data acquisition.
Once it becomes a solved problem then someone can write the gnu or open source version. I thought that was obvious? There's already an open source project.
With a throat mike, you could theoretically talk to your computer at a volume someone next to you could not hear.
I think voice interaction is inevitable - we can simply transfer information faster via vocal interactions than we can the keyboard or mouse (save for a few very fast typists or stenographers).
Ten years ago, DARPA was working on a sensor that could detect non-acoustic speech under their Advanced Speech Encoding project. Doesn't seem to have had any (publicly available) results, though.
This has been commercialized. I'm not sure if it's based on the same research, but you can find it here: www.theaudeo.com (site currently down, but it was up a couple of weeks ago).
I've been thinking about voice interfaces for home automation lately as I've been tinkering with my HA setup, so let me slightly highjack this topic: does anyone have an idea of how to set up a system where you don't have to talk directly into a microphone to get a computer to do stuff? Most current applications (e.g. voxcommando) take care of the 'speech recognition' part just fine, but I don't want to have to go hunt for my phone to turn on the light - I just want to say 'computer turn on light' and have it 'hear' me all the time. But I don't want mics in every corner either, of course.
So a sensitive mic per room, preferably one I could put deploy in severval ways (just put it down, hang at the wall, powered from mains or batteries, ...) that would send whatever it picks up to a central processing server seems the way to go.
Do such things exist? And beyond the hardware, how does one tackle e.g. the privacy problem of having everything you say recorded all the time? I'd love to have a way to say 'computer lights off' when I'm in bed just before I go to sleep, but for obvious reasons I'm hesitant to open myself up to the possibility that any dude with a high-gain antenna would be able to listen in on any sound made there...
Amazon Exho/Alexa is doing overall a good job in my house. Yes technicality it can listen to every words we say at home, but until now it didn't bother me much.
I found out recently one guy that had the time to get Amazon Alexa SDK / voices to work in conjunction with my Nest thermostat and recently SmartThings did integrate with the Echo too.
So it is now extremely convenient for asking Alexa to set the temperature by voice without the need to lookup for my phone or tablet, same in the evening to simply turn on the lights in a few places.
I do think that Voice Control in home automation is going to be a big plus.
My only gripe about the Echo is how 3rd party apps are integrated. I'd prefer it to be seamless. "Alexa, good night." should result in my HA system running a night-time scene. The current “Alexa, ask {Greeter} to {say hello world}.” is cumbersome and breaks the natural language flow. We should be able to have a conversion with the computer, not give it properly structured commands.
How sensitive is an Echo wrt positioning? What I mean is, do you have to put it in a location where you can talk in its direction or do you just put it down and even when you face away from it (let's say, from the couch), it still picks up what you say?
And does it do the voice recognition on Amazon servers or does it do that locally? Well my underlying question (since I can google technicalities like that...) is, how is the latency?
Definitely exists, but it won't be inexpensive, particularly if you want only one to cover a large room. You'll be looking for a directional microphone, with a roughly 90 degree cone. Having more than one would be a boon, the narrower the cone, the greater the range (in theory).
> how does one tackle e.g. the privacy problem
If you control the hardware doing the voice processing, it's quite simple: don't log the transcripts or save the audio files. If you can have a set of "this is a command" keywords it makes it even easier, since you can shut off the recognition while a conversation is going on, so long as it didn't start with the command keywords. If you don't control the hardware, you have to place your trust in the person who does control the hardware.
Doesn't have to be that expensive, you can get one or two cheap stereo mics per room and work on the data. Tascam's portable music/voice recorders support a very wide range of input levels.
Buy one of those cute robots in the house things.. Have the user speak to these robots instead of generally speaking into the room.. It feels a lot more natural for human beings and feels less weird since you wont have a feeling of talking to an evil bot that lives in the ether and knows everything about you.
You can use something like jasper, which is open source and can run on a raspberry pi. You don't need to have it connected to the Internet at all, you could put your home automation and the pi on its own ethernet, not connected to the rest of the world. There's nothing like an air gap.
If you had multiple microphones it might make sense to have one bulkier computer instead of having a pi in each room, but I haven't looked into how much CPU it takes jasper to listen for its wake-up word.
(thanks and I agree with you, don't get the rest of my post wrong)
Air gapping is not feasible for situations like this in 2015 imo, I mean not for 'medium security' situations like this. For example you'd want to integrate it with the rest of a home automation system, which you'd want to integrate with various online services (weather data, calendars, ip cams, not to mention access from the outside & automatic upgrades). Of course you can build all sorts of 'more secure bridges' and all, but it's just a hobby, I don't want to turn this into a job to maintain - and for mainstream roll-out, such setups are even less feasible.
I mean I understand what you say, my point is that I think that we've moved beyond that - we need to assume for applications that are not life-or-death that they will be internet-connected, and in most situations we also need to assume that they're hostile; nobody can 'know' at any point in time what is really going on in their machines and on their networks in 2015 (again, in situations where you don't have whole teams whose 24/7 job it is to monitor and be proactive).
This seems in contrast with my original question of course :) , but I was more getting at novel approaches to 'security mitigation' - which is vague to the point of useless but that's because I don't even know how to begin solving this for our current age, hence my question. Much like 10 years ago nobody knew about the blockchain and its applications, and some problems that seemed impossible to solve (not just 'technically hard', but 'impossible') maybe aren't that 'impossible' any more now.
I've devolved into rambling and I'm not expecting that someone drops the Big Revelation of the 21st century into my lap in a HN comment either, of course, so I'll just stop typing now :)
There are so many things wrong with this article. Not even mentioning that, unless I am totally screwing up as I am on 0% coffee, the Space Needle is nowhere near Washington DC.
I have a feeling we are going to head into a new era of hype and bubble that is far less defined by hardware, but more by constant promises and probably frequent disappointment about the value of virtual assistants.
A UI is basically a trade tongue. We don't speak machine and machines don't speak human.
A UI is a least common denominator that everyone can learn quickly. But it's nowhere near native quality. You can't get much more than simple one liners in trade talk. Good enough for negotiating a chest of cloves, but you'd be hard pressed to explain the difference between marxist-leninist and labour socialist perspectives on international commerce in a trade tongue.
When people hear "voice interface" what they think is a machine that can understand human so we can stop speaking goddamned pidgin all day. That is essentially a turing test, or some subspecies thereof.
I'm somewhat skeptical about this kind of thing. I'm confident that mechanical issues like translating sounds to sentences can be solved (the voice part). The translation of human to machine code… That's a different challenge.
It's an interesting case though, to have out in the wild. Siri and whatnot. An application (as opposed to an average starship computer) has a limited raison d'être, therefor a much tighter search space.
Meanwhile, we have been making consistent progress on turning our clunky pidgins into nice creoles, more elegant and powerful. Machines can understand higher level programming languages that people can get more fluent in. Average humans are generally becoming more computer literate (or fluent in UI, to keep my analogy straight). UIs are getting more intuitive, which basically means closer to human language. The common ground is expanding from just common ground and a wider language is being developed that accommodates us both better.
The question to answer is what part of that common ground is voice is open, I guess. But, I would wager that the common ground we find will not be human sounding anytime soon. It will be a UI metaphor, like the desktop. We will need to learn it.
We refer to this as the Scarlett Johanssonn problem. Any system that is smart enough to understand what we mean by the Scarlett Johanssonn problem, and the reference to Artificially Intelligent virtual assistants, would itself need to be a strong AI.
We've already solved the problem of translating human to machine code for graphical user interfaces. It's the desktop with the keyboard and mouse. It's the app and the touchscreen.
You're exactly right that the question now is how can we do the same translation for voice-based systems.
So far the approach has been Virtual assistants, which is really hard, see again the Scarlett Johanssonn problem.
We've been working to find the right design/interaction metaphor, and the app metaphor seems to work really well.
Not only is the search space limited, like you mentioned, but users have a more intuitive since of how to interact with the app. When you open a shopping app, you know that you can likely search for something to buy ("I'm looking for a new pair of shoes"). When you interact with a general virtual assistant, that interaction is less clear.
The app model also works for scaling conversational interaction beyond a single context. Individual apps can support their own language for interaction which best suits their function. This of course necessitates design standards, just like we have for GUIs.
The limitations for fully voice-based applications aren't technical at this point. They are design problems with known solutions. The break out for voice will happen when the right form factor comes about with the right UI design.
>The break out for voice will happen when the right form factor comes about with the right UI design.
Chris Maury? If so, hoping to see your progress on this with Conversant Labs in the coming year or so. After I read the article I was wondering if I'd see you on HN.
Also my daughter (who also has Stargardt's) found it very inspiring to chat with you at the accessibility meetup.
You caught me :). It was great meeting you daughter! She seems to be doing great and taking things on with the right attitude.
Feel free to email me anytime, if you ever need anything: Chris@Conversantlabs.com. Happy to talk through things in a more private setting than the comments section of HN.
This is a really well put explanation of why the natural language processing side is hard, but it assumes that we can even get letters and words into the system.
I'm a linguist who works on speech perception, and I'm increasingly convinced that true speech-to-text, that can stand in for a human transcriptionist, is an AI-complete problem. We can get close, but you'll always need context and understanding of the world to accurately get things like "I'm going to take a walk in the park, then I'm going to take a wok from the Chinese Restaurant".
Siri and whatnot are amazing, and there's room for improvement, especially on the NLP end, but these issues combined with how damned hard speech is mean that Jarvis will be a keyboard away for years to come.
I had a worrying discussion the other day with somebody about user interfaces. I was asserting that UIs are more than just CLIs and GUIs, more than VUIs (Voice User Interface)...UIs extend to virtually everything we do to interact with a computer. A programming language is a kind of UI, an API is a kind of UI and so on.
I was asserting that when we think of the entire interaction surface of a computer, from bare metal on up, as an interface (with us as the users), then we need to make sure we design those interfaces to work well as an interaction medium because it enables "better" interaction with machines (on virtually every vector).
"The" native quality is complete brain-computer merging.
Brain-computer direct communication is close but not quite there.
It's not 100% until your thoughts are completely merged with the computer's. Your thought can't be called your own anymore, and it can't be called the computer's thought. At this point, you and the machine have become one, and latency has been reduced to zero.
Until I can have a conversation with a machine–like I would a person–then I feel like voice interfaces will languish in the uncanny valley of HCI. I notice the problem the most when listening to voice commands from a GPS; there are times when the computer gives me an instruction, and I really want to say , "That's not going to happen."
It would be nice if it could pre-emptively start to recompute–like a person navigator would–rather than wait until the GPS detects that I'm not on the course that it thinks I'm going to be on to make a correction that will essentially go the same way. But we can't have a _conversation_; just an exchange of commands. The machine tells the meatbag to do something, the meatbag tells the machine to do something.
Tangentially, I don't _really_ want voice interfaces like in Star Trek/Jarvis; I want a digital doppelgänger. Something that can act on my behalf and convincingly sound/act/respond like I would. I don't want to talk to a machines more, I want to talk to other humans less.
33 comments
[ 4.5 ms ] story [ 78.7 ms ] threadI really don't want to talk to my computer/phone/coffee pot/house. I would rather that software have its normal interface polished and made closer to something that is usable, than have another UI paradigm bolted on that is going to be badly implemented 80% of the time by application developers.
But, give it a little time and you can expect that authenticated users will be getting speaker adaptation.
"We were promised Jarvis, but all we got were eight different closed-source, non-extensible, hosted voice recognition platforms."
What are the leading voice recognition assistant thingies? Siri, Google Now (no human name, much like most google employees), Cortana, Alexis? They each have huge acoustic models and run completely privately and are completely closed source.
We can't have Jarvis until we run our own hackable computing infrastructures personally, not globally hosted versions running behind the walls of Trust Us, Inc.
Which mobile platform will be the first to offer pluggable voice-to-action backends? Much like we pick web browsers, we should be able to select our voice command interfaces, but that would require a lot more interop between hardware manufactures and would also kill their moats of unlimited training data acquisition.
https://en.m.wikipedia.org/wiki/CMU_Sphinx
The others are leading because they are putting more time and money into their respective projects.
I think voice interaction is inevitable - we can simply transfer information faster via vocal interactions than we can the keyboard or mouse (save for a few very fast typists or stenographers).
[1] https://web.archive.org/web/20150330030801/http://www.theaud...
So a sensitive mic per room, preferably one I could put deploy in severval ways (just put it down, hang at the wall, powered from mains or batteries, ...) that would send whatever it picks up to a central processing server seems the way to go.
Do such things exist? And beyond the hardware, how does one tackle e.g. the privacy problem of having everything you say recorded all the time? I'd love to have a way to say 'computer lights off' when I'm in bed just before I go to sleep, but for obvious reasons I'm hesitant to open myself up to the possibility that any dude with a high-gain antenna would be able to listen in on any sound made there...
And does it do the voice recognition on Amazon servers or does it do that locally? Well my underlying question (since I can google technicalities like that...) is, how is the latency?
Definitely exists, but it won't be inexpensive, particularly if you want only one to cover a large room. You'll be looking for a directional microphone, with a roughly 90 degree cone. Having more than one would be a boon, the narrower the cone, the greater the range (in theory).
> how does one tackle e.g. the privacy problem
If you control the hardware doing the voice processing, it's quite simple: don't log the transcripts or save the audio files. If you can have a set of "this is a command" keywords it makes it even easier, since you can shut off the recognition while a conversation is going on, so long as it didn't start with the command keywords. If you don't control the hardware, you have to place your trust in the person who does control the hardware.
Buy one of those cute robots in the house things.. Have the user speak to these robots instead of generally speaking into the room.. It feels a lot more natural for human beings and feels less weird since you wont have a feeling of talking to an evil bot that lives in the ether and knows everything about you.
If you had multiple microphones it might make sense to have one bulkier computer instead of having a pi in each room, but I haven't looked into how much CPU it takes jasper to listen for its wake-up word.
https://jasperproject.github.io/
Air gapping is not feasible for situations like this in 2015 imo, I mean not for 'medium security' situations like this. For example you'd want to integrate it with the rest of a home automation system, which you'd want to integrate with various online services (weather data, calendars, ip cams, not to mention access from the outside & automatic upgrades). Of course you can build all sorts of 'more secure bridges' and all, but it's just a hobby, I don't want to turn this into a job to maintain - and for mainstream roll-out, such setups are even less feasible.
I mean I understand what you say, my point is that I think that we've moved beyond that - we need to assume for applications that are not life-or-death that they will be internet-connected, and in most situations we also need to assume that they're hostile; nobody can 'know' at any point in time what is really going on in their machines and on their networks in 2015 (again, in situations where you don't have whole teams whose 24/7 job it is to monitor and be proactive).
This seems in contrast with my original question of course :) , but I was more getting at novel approaches to 'security mitigation' - which is vague to the point of useless but that's because I don't even know how to begin solving this for our current age, hence my question. Much like 10 years ago nobody knew about the blockchain and its applications, and some problems that seemed impossible to solve (not just 'technically hard', but 'impossible') maybe aren't that 'impossible' any more now.
I've devolved into rambling and I'm not expecting that someone drops the Big Revelation of the 21st century into my lap in a HN comment either, of course, so I'll just stop typing now :)
I have a feeling we are going to head into a new era of hype and bubble that is far less defined by hardware, but more by constant promises and probably frequent disappointment about the value of virtual assistants.
A UI is a least common denominator that everyone can learn quickly. But it's nowhere near native quality. You can't get much more than simple one liners in trade talk. Good enough for negotiating a chest of cloves, but you'd be hard pressed to explain the difference between marxist-leninist and labour socialist perspectives on international commerce in a trade tongue.
When people hear "voice interface" what they think is a machine that can understand human so we can stop speaking goddamned pidgin all day. That is essentially a turing test, or some subspecies thereof.
I'm somewhat skeptical about this kind of thing. I'm confident that mechanical issues like translating sounds to sentences can be solved (the voice part). The translation of human to machine code… That's a different challenge.
It's an interesting case though, to have out in the wild. Siri and whatnot. An application (as opposed to an average starship computer) has a limited raison d'être, therefor a much tighter search space.
Meanwhile, we have been making consistent progress on turning our clunky pidgins into nice creoles, more elegant and powerful. Machines can understand higher level programming languages that people can get more fluent in. Average humans are generally becoming more computer literate (or fluent in UI, to keep my analogy straight). UIs are getting more intuitive, which basically means closer to human language. The common ground is expanding from just common ground and a wider language is being developed that accommodates us both better.
The question to answer is what part of that common ground is voice is open, I guess. But, I would wager that the common ground we find will not be human sounding anytime soon. It will be a UI metaphor, like the desktop. We will need to learn it.
We refer to this as the Scarlett Johanssonn problem. Any system that is smart enough to understand what we mean by the Scarlett Johanssonn problem, and the reference to Artificially Intelligent virtual assistants, would itself need to be a strong AI.
We've already solved the problem of translating human to machine code for graphical user interfaces. It's the desktop with the keyboard and mouse. It's the app and the touchscreen.
You're exactly right that the question now is how can we do the same translation for voice-based systems.
So far the approach has been Virtual assistants, which is really hard, see again the Scarlett Johanssonn problem.
We've been working to find the right design/interaction metaphor, and the app metaphor seems to work really well.
Not only is the search space limited, like you mentioned, but users have a more intuitive since of how to interact with the app. When you open a shopping app, you know that you can likely search for something to buy ("I'm looking for a new pair of shoes"). When you interact with a general virtual assistant, that interaction is less clear.
The app model also works for scaling conversational interaction beyond a single context. Individual apps can support their own language for interaction which best suits their function. This of course necessitates design standards, just like we have for GUIs.
The limitations for fully voice-based applications aren't technical at this point. They are design problems with known solutions. The break out for voice will happen when the right form factor comes about with the right UI design.
Chris Maury? If so, hoping to see your progress on this with Conversant Labs in the coming year or so. After I read the article I was wondering if I'd see you on HN.
Also my daughter (who also has Stargardt's) found it very inspiring to chat with you at the accessibility meetup.
Feel free to email me anytime, if you ever need anything: Chris@Conversantlabs.com. Happy to talk through things in a more private setting than the comments section of HN.
I'm a linguist who works on speech perception, and I'm increasingly convinced that true speech-to-text, that can stand in for a human transcriptionist, is an AI-complete problem. We can get close, but you'll always need context and understanding of the world to accurately get things like "I'm going to take a walk in the park, then I'm going to take a wok from the Chinese Restaurant".
Siri and whatnot are amazing, and there's room for improvement, especially on the NLP end, but these issues combined with how damned hard speech is mean that Jarvis will be a keyboard away for years to come.
I had a worrying discussion the other day with somebody about user interfaces. I was asserting that UIs are more than just CLIs and GUIs, more than VUIs (Voice User Interface)...UIs extend to virtually everything we do to interact with a computer. A programming language is a kind of UI, an API is a kind of UI and so on.
I was asserting that when we think of the entire interaction surface of a computer, from bare metal on up, as an interface (with us as the users), then we need to make sure we design those interfaces to work well as an interaction medium because it enables "better" interaction with machines (on virtually every vector).
Brain-computer direct communication is close but not quite there.
It's not 100% until your thoughts are completely merged with the computer's. Your thought can't be called your own anymore, and it can't be called the computer's thought. At this point, you and the machine have become one, and latency has been reduced to zero.
It would be nice if it could pre-emptively start to recompute–like a person navigator would–rather than wait until the GPS detects that I'm not on the course that it thinks I'm going to be on to make a correction that will essentially go the same way. But we can't have a _conversation_; just an exchange of commands. The machine tells the meatbag to do something, the meatbag tells the machine to do something.
Tangentially, I don't _really_ want voice interfaces like in Star Trek/Jarvis; I want a digital doppelgänger. Something that can act on my behalf and convincingly sound/act/respond like I would. I don't want to talk to a machines more, I want to talk to other humans less.