I always find these cool, and then remember that I know someone who built a working (well working!) prototype of this in a weekend no almost no fanfare, from practically scratch.
with the caveat of not knowing your that someone in any way shape or form...
These are students, young/new engineers. This is impressive and should be lauded as should all attempts to push what an individual person is capable of. It may not be new or novel to everyone, but it is to them...basically, cut them a bit of slack.
The people I'm talking about were also students, (University Hackathon). A group of four, I want to say 2 sophomores and 2 freshmen.
This was 2-3 years ago, so machine learning wasn't advanced (and even now, getting a functional NN running and trained in a weekend isn't feasible) so they only had it recognize a small set of commands, but it could do that.
If you're getting RSI from sign language you're either an interpreter or over-producing.
In the case of the former, you're using the language differently than a typical user. Most people don't produce language in the same amounts or in the same ways that an interpreter will (that is; speaking/signing isn't most people's job). There are steps you can take to reduce RSI as an interpreter, including proper rest while working and supportive braces at night.
For the latter, it's possible to over-produce signs - basically "signing too hard." That can be painful, but it's correctable.
> Most people don't produce language in the same amounts that an interpreter will
It isn't really because of how much they sign (deaf people don't typically get RSI from sign language), but rather because of how they sign it. The signing is more precise and formal, the hands/arms are elevated more, and generally less "colloquial" than you would see in day-to-day sign language.
Volume of signing isn't totally unrelated - even if you presume 4 hours of signing time per day, most people don't speak/sign for that long an amount of time. It's more than a typical deaf person, any way you slice it.
But, you're right - the manner of signing has something to do with it as well, and I've amended my comment to reflect that. :)
This is a really novel approach, but IMHO a more well rounded solution is the one being developed at Microsoft Research using Kinect. https://www.youtube.com/watch?v=HnkQyUo3134
Nonetheless, great work by these students. I think their work is likely to have more applications in other areas where mimicking the movement of hands and fingers is an essential component.
Don't know why you got downvoted. Your comment is both admirably terse, and completely valid!
SL is an entirely distinct language. People who are fluent in multiple languages switch from linguistic context to another when they speak different languages. They don't translate from their native language. So it makes sense that you don't do this either.
To properly translate ASL, you need to pick up a significant amount of grammar from the face (called non-manual markers) as well as body position, 3d capture of spatial relationships between signs as they are made, and other features which would be inaccessible to these sensors.
And then there's the fact that ASL isn't serialized English (it's actually unrelated to English at all, and has its roots in the French Sign Language). There's no one-to-one mapping between ASL signs and English words, just as there's no one-to-one mapping between Chinese and French, or any two languages. And ASL has grammar features not seen in any spoken language, to complicate things further.
These sensors could perhaps be used to translate other sign languages, but I'm really skeptical.
I wasn't aware of the existence of any notation at all! The dictionary I have uses photographs, other dictionaries and textbooks use drawings of people speaking SL. The nearest thing to a notation in this case consists of a verbal description of face and hand gestures as they are carried out in real time.
I have often wondered if it would be possible to create a notation using the simplest and most abstract rendition of an actual SL speaker's signing. Lines and curves that hold the semantic essence of the movement of face, hand and posture. It would take a native speaker to be able to abstract what is truly essential to meaning, and a community to agree on its elegance and clarity.
Off topic, but connected: I discovered from experience that small children can form the SL alphabet with their hands before they acquire the coordination to draw written letters. I found it very useful in teaching my children to read.
Correction to my comment above: On taking out my Dictionary of British Sign Language, I find that there is indeed an intricate notation used throughout.
Yes it matters. I didn't say anything about a standard. I don't mean that "there isn't an agreed upon notation that Deaf Americans use to write ASL", I mean "we don't have a good way represent signs that allows us to search with that representation, apply phonological rules, and synthesize signs from that representation".
We have all sorts of representations for spoken speech---a waveform, spectrogram, vocal tract models, IPA--and these all get used in a speech recognition (or synthesis) system. For ASL we have movies, still-image cartoons that depict motion, categories for some parameters like handshape, some linguistic insights, and I'm not sure what else.
Very cool, but it's not quite the revolution described by the inventors in the video - isn't it only one-way, assuming a lot of the deaf person's lip-reading ability?
I'm not sure we understand each other. If the signing person can speak to employers etc. now via the device, they can only speak back with their mouths, or possibly speech recognition if well set up. The deaf person would have to be able to lip-read people who potentially don't know how to help them in that area. In other words, it might still be tricky to communicate.
Seems cool, but I wonder how much of this is faked in the demo video. The man signs "T" and it's translated to "Thomas", "N" to "Navid", and "L" to "Lemelson". So unless this is a learning system as well, I'm skeptical. Lot of potential here for sure though, hope they keep it up.
I wonder if they've just hard coded their names for now because they had trouble doing fingerspelling, because I doubt that someone's name-sign is just a 'T'. That's really close to the sign for 'bathroom', which would be weird :)
Could be a home-sign, or a common shortcut. If two people are discussing two people who aren't physically present, using just their first initials could be a very obvious way of referring to them unambiguously without having to sign their full names.
1) This is an informal shorthand agreed between people who already know each other in the particular context of work, so there is no surprise or ambiguity. Outside this context, I would use, and expect to see the dictionary sign.
2) It functions as an 'in-group' slang, boosting cohesion and good will.
3) There is a certain humorousness in bending a symbol into an informal use. This is more pronounced in cases where the alternate meaning is irreverent or vulgar. I have no intention of expanding on the second point.
4) One does not have to be facing the SL speaker. The 't' symbol can be recognized from beside the person or even behind. Point 5 gives the rationale for this.
5) It can be communicated without changing focus or posture, allowing concentration on the task at hand without breaking the workflow.
6) Though 'thanks' is a brief gesture, 't' can be signed and understood even more quickly. Just a flash of meaning.
7) The sign for 'good', while it is also used to communicate thanks and has advantages 4,5,6 listed above, can be ambiguous, violating the principle of least surprise (see point 1 above). It is more generic, while 't' is specific.
However, I think it's fair to say that a rudimentary system like this one is scripted, and is not tuned to the informal and complicated contexts you describe. You can argue that it's fair game to pre-program the device to emit "Lemelson" for #L, just as you might prepare an interpreter with proper nouns before a gig, but I think it goes against the spirit of the claims made in the video.
some friends of mine translate sign language to text/speech using leap motion and/or kinect to figure out the best way for it. there is a startup that uses leap motion and an ipad but it looks very clunky to use. my friends are connecting to the user's smartphone for the translation work.
It's a great project, but to nail this they are going to need more involvement from the deaf community as they've missed some key points:
1)ASL isn't just about hand movements, there is a ton of grammar that comes from facial expression. It's not just like English tone, but can actually change a word. Like can become "don't like" from facial expression.
2) Context is very key, there are a lot of words with multiple meanings that matter based on the sentence. It could be figured out from a really smart system but wouldn't be easy.
3) Lastly, ASL isn't English. This is the biggest mistake people make. You aren't simply putting words into symbols. ASL has it's own grammar, syntax and even slang. I remember spending 15 minutes trying to figure out what my ASL teacher was saying when he said he "ate champ burger" and it turns out he meant it was the best burger he ever had. To 'translate' ASL you need to move from symbols to words and then add in the context.
Ultimately there is a reason that professionals in this industry are called "sign language interpreters" and not "sign language translators" ASL is really hard to put into spoken or written language.
It's also worth noting at really high level events an ASL interpreter will translate spoken word into sign and then a "Certified Deaf Interpreter" will re-sign it because the ASL of someone who is has communicated in sign their entire life has far more facial expression and meaning than even professional interpreters who are hearing.
I think it's great to see advancements in technology trying to assist the deaf community, but I think it's important to get early involvement rather than assume we know what people need.
I hope they keep up the great work - it's on the right path.
Actually, the distinction (that I've seen) between "translator" and "interpreter" is whether the "transformation" is happening in real-time across spoken/sign language, or whether it's written language. Translators take text in one language and write it in another, like translating a legal document. Interpreters consume speech/sign in one language and speak/sign it in another.
With all the critiques in the comments so far, it is worth noting that this article simply says they won a contest for being promising inventors, not that the invention doesn't have flaws.
47 comments
[ 0.23 ms ] story [ 99.8 ms ] threadThese are students, young/new engineers. This is impressive and should be lauded as should all attempts to push what an individual person is capable of. It may not be new or novel to everyone, but it is to them...basically, cut them a bit of slack.
This was 2-3 years ago, so machine learning wasn't advanced (and even now, getting a functional NN running and trained in a weekend isn't feasible) so they only had it recognize a small set of commands, but it could do that.
I've never heard of a case where a casual user gets RSI.
http://www.youtube.com/watch?v=0QNiZfSsPc0
There are others: https://github.com/melling/ErgonomicNotes#gesture-computing
In the case of the former, you're using the language differently than a typical user. Most people don't produce language in the same amounts or in the same ways that an interpreter will (that is; speaking/signing isn't most people's job). There are steps you can take to reduce RSI as an interpreter, including proper rest while working and supportive braces at night.
For the latter, it's possible to over-produce signs - basically "signing too hard." That can be painful, but it's correctable.
It isn't really because of how much they sign (deaf people don't typically get RSI from sign language), but rather because of how they sign it. The signing is more precise and formal, the hands/arms are elevated more, and generally less "colloquial" than you would see in day-to-day sign language.
But, you're right - the manner of signing has something to do with it as well, and I've amended my comment to reflect that. :)
Nonetheless, great work by these students. I think their work is likely to have more applications in other areas where mimicking the movement of hands and fingers is an essential component.
https://en.wikipedia.org/wiki/Chorded_keyboard
SL is an entirely distinct language. People who are fluent in multiple languages switch from linguistic context to another when they speak different languages. They don't translate from their native language. So it makes sense that you don't do this either.
And then there's the fact that ASL isn't serialized English (it's actually unrelated to English at all, and has its roots in the French Sign Language). There's no one-to-one mapping between ASL signs and English words, just as there's no one-to-one mapping between Chinese and French, or any two languages. And ASL has grammar features not seen in any spoken language, to complicate things further.
These sensors could perhaps be used to translate other sign languages, but I'm really skeptical.
I have often wondered if it would be possible to create a notation using the simplest and most abstract rendition of an actual SL speaker's signing. Lines and curves that hold the semantic essence of the movement of face, hand and posture. It would take a native speaker to be able to abstract what is truly essential to meaning, and a community to agree on its elegance and clarity.
Off topic, but connected: I discovered from experience that small children can form the SL alphabet with their hands before they acquire the coordination to draw written letters. I found it very useful in teaching my children to read.
We have all sorts of representations for spoken speech---a waveform, spectrogram, vocal tract models, IPA--and these all get used in a speech recognition (or synthesis) system. For ASL we have movies, still-image cartoons that depict motion, categories for some parameters like handshape, some linguistic insights, and I'm not sure what else.
1) This is an informal shorthand agreed between people who already know each other in the particular context of work, so there is no surprise or ambiguity. Outside this context, I would use, and expect to see the dictionary sign.
2) It functions as an 'in-group' slang, boosting cohesion and good will.
3) There is a certain humorousness in bending a symbol into an informal use. This is more pronounced in cases where the alternate meaning is irreverent or vulgar. I have no intention of expanding on the second point.
4) One does not have to be facing the SL speaker. The 't' symbol can be recognized from beside the person or even behind. Point 5 gives the rationale for this.
5) It can be communicated without changing focus or posture, allowing concentration on the task at hand without breaking the workflow.
6) Though 'thanks' is a brief gesture, 't' can be signed and understood even more quickly. Just a flash of meaning.
7) The sign for 'good', while it is also used to communicate thanks and has advantages 4,5,6 listed above, can be ambiguous, violating the principle of least surprise (see point 1 above). It is more generic, while 't' is specific.
However, I think it's fair to say that a rudimentary system like this one is scripted, and is not tuned to the informal and complicated contexts you describe. You can argue that it's fair game to pre-program the device to emit "Lemelson" for #L, just as you might prepare an interpreter with proper nouns before a gig, but I think it goes against the spirit of the claims made in the video.
1)ASL isn't just about hand movements, there is a ton of grammar that comes from facial expression. It's not just like English tone, but can actually change a word. Like can become "don't like" from facial expression.
2) Context is very key, there are a lot of words with multiple meanings that matter based on the sentence. It could be figured out from a really smart system but wouldn't be easy.
3) Lastly, ASL isn't English. This is the biggest mistake people make. You aren't simply putting words into symbols. ASL has it's own grammar, syntax and even slang. I remember spending 15 minutes trying to figure out what my ASL teacher was saying when he said he "ate champ burger" and it turns out he meant it was the best burger he ever had. To 'translate' ASL you need to move from symbols to words and then add in the context.
Ultimately there is a reason that professionals in this industry are called "sign language interpreters" and not "sign language translators" ASL is really hard to put into spoken or written language.
It's also worth noting at really high level events an ASL interpreter will translate spoken word into sign and then a "Certified Deaf Interpreter" will re-sign it because the ASL of someone who is has communicated in sign their entire life has far more facial expression and meaning than even professional interpreters who are hearing.
I think it's great to see advancements in technology trying to assist the deaf community, but I think it's important to get early involvement rather than assume we know what people need.
I hope they keep up the great work - it's on the right path.
http://katies.online/katiesblog/index.php/2016/04/27/yes-ive...
http://www.washington.edu/news/2016/04/12/uw-undergraduate-t...