I love them doing accessibility stuff, and I can see this being useful for specific people in my life. But when I initially read the announcement I thought about using it to transcribe business meetings.
Obviously the accuracy of this won't be "court reporter"-levels, but for casual note taking it would be "good enough."
Google Docs has a voice typing functionality [1]. I used Soundflower [2] to send audio playback to a digital input device, and had Google Docs listen to this device. This worked "okay" for transcribing a blog post from a voice memo.
I think it has some privacy implications. Lot of people will feel uncomfortable if they feel their conversations are being saved in someone else's device or cloud.
how long until we start seeing computer vision applications that translate sign language? It's a pretty hard problem, but definitely something that's feasible with the right methods.
Wouldn't this be used in the same way as with any other language? I.e. customer only knows english sign language, but I don't, luckily my cash-register has a camera that can translate sign language to my language?
Although there is a lot of hype around AI/ML, there are a ton of pragmatic things coming out of the research much like this Live Transcription by Google that I think will be incredibly helpful to people around the world.
The thing I am most excited about is that most of this work is being done in the open, in the very least much of this is being open sourced by Google, Facebook, and other giants. For all the heat they have been taking lately I do think they deserve to be applauded for this and Of course this is mostly happening in order to sell us cloud services but it cannot be understated how helpful this will be to many around the world.
Another thing I’m excited about is what I can build with these things, and even more what others will create with these models as building blocks in their applications.
As an example, and complete self promotion here, I was able to use some open source models by Google on TensorFlow to build a cross platform App that can read Articles to you using these neural networks. The amazing thing is I built it mostly on nights and weekends, which shows how easy some of this is to work with now, you can check it out here if you like https://articulu.com
Google's ASR (automatic speech recognition) is conducted on their Cloud infrastructure. In fact almost everyone's ASR is done online because the database of speech profiles is huge. Big data is the secret sauce that makes modern ASR superior to previous attempts.
This app is cool and useful, but a major piece of the puzzle is missing.
Communication between a Deaf person and a hearing person is a two-way street, and this tool really only addresses one of those streets.
The tool transcribes the audible speech of the hearing person, allowing the Deaf person to read the transcription.
But if the Deaf person wants to sign a response, they're out of luck. Instead, they need to type their response on the device.
That's OK, but in a "this'll make do in a pinch" kind of way. The ideal is that both the Deaf person and the Hearing person are able to communicate without typing.
There is some pretty cool research around signing-to-text translation -- Matt Huenerfauth https://huenerfauth.ist.rit.edu is doing some really interesting stuff, for example -- but as far as I know it's not ready for prime time.
As a deaf person, I use https://www.ava.me/ most of the time. It allows me to host a group conversation and have speakers labeled if everyone else uses the app but that's a chore to set up sometimes. It's also quite pricey at $30 per month and some months I use it for less than a hour. It can voice what you type if you want, but I normally voice for myself. I also used to use Keep Notes & the 'mic' icon in a pinch, but that would turn itself off if it didn't pick up any speech in some time.
There is a keyboard button on the toolbar (portrait mode). When you press it, a keyboard shows up so that the user can communicate back to the speaker.
I read about it and tried it out, and I couldn't figure out how to use it. After typing my text, I kept looking for the button to trigger text to speech.
But, it just dawned on me that the idea is to type something and then show it to the other person. Which works, it just wasn't what I was expecting.
> This app is cool and useful, but a major piece of the puzzle is missing
While I agree pieces are missing when sign communication is needed, just this piece can still life changing for the hard of hearing (who vastly outnumber the deaf). This is for my mother in-law who stopped going to group lunches because she felt so left out, even with hearing aids. If this app works as advertised, there are going to be tablets mounted on the walls running it. And maybe a phone adapter.
now they just need to bolt it onto Glass or some other HUD. IRL subtitles. Could put a mic array on the frame too, and then get an indication of where the speaker is so you can be alerted even if you aren't facing the person.
So GPDR limits having your Google Home device listen in all the time because, well that is creepy, but maybe you want a transcript so let us listen in on your conversations please.
Given that the service is free, you have to know that Google has a plan to monetize that data with someone you don't know yet. When you find out who that is you won't be able to 'undo' the transcriptions you have done in the past.
I also think it is ridiculous that this needs "the cloud". I had pretty good speaker dependent transcription working on an Intel 486 processor, and so find it difficult to believe that given literally 500x the compute power and 2000x the memory on a typical phone you can't do this all locally?
32 comments
[ 0.26 ms ] story [ 79.3 ms ] threadI love them doing accessibility stuff, and I can see this being useful for specific people in my life. But when I initially read the announcement I thought about using it to transcribe business meetings.
Obviously the accuracy of this won't be "court reporter"-levels, but for casual note taking it would be "good enough."
[1] https://qz.com/work/1087765/how-to-transcribe-audio-fast-and... [2] https://github.com/mattingalls/Soundflower
I'll add that I'd feel MORE comfortable with just a transcript being saved than I would an audio file (which is what happens now). Deniability.
Auto-transcribing likely falls in a grey zone, because you could argue that it is being recorded before transcription (which is true).
The thing I am most excited about is that most of this work is being done in the open, in the very least much of this is being open sourced by Google, Facebook, and other giants. For all the heat they have been taking lately I do think they deserve to be applauded for this and Of course this is mostly happening in order to sell us cloud services but it cannot be understated how helpful this will be to many around the world.
Another thing I’m excited about is what I can build with these things, and even more what others will create with these models as building blocks in their applications.
As an example, and complete self promotion here, I was able to use some open source models by Google on TensorFlow to build a cross platform App that can read Articles to you using these neural networks. The amazing thing is I built it mostly on nights and weekends, which shows how easy some of this is to work with now, you can check it out here if you like https://articulu.com
Communication between a Deaf person and a hearing person is a two-way street, and this tool really only addresses one of those streets.
The tool transcribes the audible speech of the hearing person, allowing the Deaf person to read the transcription.
But if the Deaf person wants to sign a response, they're out of luck. Instead, they need to type their response on the device.
That's OK, but in a "this'll make do in a pinch" kind of way. The ideal is that both the Deaf person and the Hearing person are able to communicate without typing.
There is some pretty cool research around signing-to-text translation -- Matt Huenerfauth https://huenerfauth.ist.rit.edu is doing some really interesting stuff, for example -- but as far as I know it's not ready for prime time.
Could have not said it better myself.
But, it just dawned on me that the idea is to type something and then show it to the other person. Which works, it just wasn't what I was expecting.
While I agree pieces are missing when sign communication is needed, just this piece can still life changing for the hard of hearing (who vastly outnumber the deaf). This is for my mother in-law who stopped going to group lunches because she felt so left out, even with hearing aids. If this app works as advertised, there are going to be tablets mounted on the walls running it. And maybe a phone adapter.
https://www.youtube.com/watch?v=zL6ltnSKf9k
Have the HUD have a dot. When you're wearing the glasses, you aim the red dot on the speaker. The transcription could come just from that speaker.
Just a thought.
Given that the service is free, you have to know that Google has a plan to monetize that data with someone you don't know yet. When you find out who that is you won't be able to 'undo' the transcriptions you have done in the past.
I also think it is ridiculous that this needs "the cloud". I had pretty good speaker dependent transcription working on an Intel 486 processor, and so find it difficult to believe that given literally 500x the compute power and 2000x the memory on a typical phone you can't do this all locally?