I always wondered how someone could use text-to-speech effectively, especially with a programming language full of symbols with long names such as 'semi-colon', 'underscore', 'left parenthesis' etc. The answer it seems is to speed up the voice tenfold! Honestly incredible he's fine-tuned his other senses well enough to be able to code at a reasonable speed again.
> The answer it seems is to speed up the voice tenfold!
That seems very common for systematic text-to-speech users, I saw a video a few months back of a lawyer (?) demonstrating ios's accessibility (rotor, voiceover, blind curtain, …) and the TTS speech was so fast it sounded like white noise to me.
I was at a web design conference once with a talk by a heavily vision-impaired guy. I was amazed at the speed when he went to his IDE to fix a bug. I couldn't make out a word it was saying and he didn't seem slower working with it than I would be.
I'm glad you mentioned that. After watching this it made me wonder if this is normal, or if this guy is some sort of auditory wizard. It seems like an awful lot of work to master, but maybe not if it's a skill you're using everyday.
> After watching this it made me wonder if this is normal, or if this guy is some sort of auditory wizard.
The "visual cortex" area is not deactivated but gets re-tasked with non-visual processing tasks, even in "late blind" people (people who lost their sight after age 12): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3667661/
Considering the amount of visual information the sighted process more or less effortlessly, that's a pretty significant amount of processing capabilities which can get retasked with e.g. auditory processing.
One of the company I worked with had to design an audio interface for blind people. I proposed something relying on several stereo located sounds of different pitches and intensity. I was told this would be too hard to use for a normal user. I wish I had such a video at the time to show the kind of bandwidth a blind person is able to get from audio alone.
Oh, that one hanging out at the end of the line? Hopefully, a better assistant could suggest that as a fix then you'd simply have to say "fix line 11", for example. It has got to get increasingly difficult to find even simple syntax errors in a lot of code, if you are visually impaired.
I know it's no where near the same thing. But this reminds me of when I broke my arm very bad as a kid, and was in a cast without the use of my fingers. I learned how to play Mario Brothers and Zelda with just my left hand in a few weeks. Besides a few key combinations, like working up speed in Mario, I ended up being better with one hand than two.
I imagine many would benefit if we had a smart developer assistant where it could have a higher level understanding of programming.
"Jarvis import library foo" - could be said from anywhere in file
"Jarvis jump to file bar.c"
"Jarvis declare var current temperature equals zero" - var currentTemperature = 0
"Jarvis write for enumeration on var word over var name list"
"Jarvis build and run" or simply "Jarvis run"
By the way, I know many people don't like to talk to a computer, can't imagine how it could be faster than typing, etc. However, I believe that once an assistant is done right, more than a few people would be converts.
I'm sure someone on HN is seeing this right now and has already started working on it :). Especially with the advent of Google Assistant SDK and the likes, it's clearly very much possible now.
IMHO I don't think it would fly. The most difficult things of programming are too abstract for that.
You may be interested in the guy that did all his coding with voice for a while because he couldn't use his hands, and still uses it to this day about half of the time. https://www.youtube.com/watch?v=8SkdfdXWYaI
If I'm reading or thinking and leaning back in my chair I often want to say or gesture a command to scroll up or down for example, it happens more often than not when reading long articles.
(btw, if anyone knows anything about software/hardware to do that? let me know)
It would good to have the options to voice or gesture command a desktop computer.
I used to do this with Dragon Dictate. It worked very well; I recommend it.
I created speech bindings for my most common navigation tasks:
- "Down" -> presses PgDn
- "Down a bit" -> presses "down" a few times
- "Switch" -> presses Cmd+Tab (next application)
- "Backtick" -> presses Cmd+` (next window)
- "Apple L" -> presses Cmd+L (focus address bar)
Latency was good enough. Bindings have a higher priority than dictation, so if it hears one of your custom bindings: it will immediately action it instead of waiting to hear a full sentence.
Mac version was mostly only reliable for triggering keyboard shortcuts. Dictation of sentences worked pretty well, but there were (many) significant bugs that meant it was not generally viable.
I eventually stopped because of one of the following (can't recall):
- OS updates broke it
- lost the license in a computer switch
- hands got better
Yep, there are demands you can certainly make after the problem is solved.
For now, it would be helpful to many if there were no such limitations. In other words, comments like "Make it even smarter and it can't go to the cloud then I might use it" can safely be ignored for now. There's a market for an MVP that does a lot less.
Rather than my usual tirade against open offices, I would point out that options for subvocal voice pickups exist like throat mikes. You could speak softly enough your neighbor couldn't hear you, and they also attenuate external sounds quite well.
So basically you'll be trying to think above a constant, low-level din of unintelligible murmuring. Not unlike having a brown noise generator in your office. That would drive me insane.
Well, when I say "your neighbor can not hear you", I mean that literally. Not mumbling, just talking at a sub-audible volume.
And, IMO, even mumbling would be better than the current open office constant flow of loud conversations (one of which I'm trying to drown out with headphones this very moment).
Voice input is an option when other types of input are not.
In the same vein, I am learning morse code. Any resources helping with this would be appreciated.
I would like to use morse code for silent, no-look I/O on my phone. I'm not sure vibrations can be quiet enough; might have to use something special-purpose plugged into the headphone jack.
And then one day there will be a generation of programmers who have only known coding via a voice-driven developer assistant. Imagine trying to interview such a person.
"Please stop shouting at the whiteboard. It doesn't work that way. I need you to pick up one of the dry-erase markers and write your algorithm to invert a binary tree."
"What? Who codes with their hands? I need my assistant!"
Would that be a bad thing though ? Assuming assistants are ubiquitous.
We don't use punch cards in interviews today.
Even today whiteboard interviews are sometimes not the best method to fill certain roles.
Listen Mr. Senior Principal Architect, not only cheap H1B Indian guy, but even cheap blind H1B Indian guy can replace you!
What I don't like about MS is that in their promo materials developing software is so easy. Drag and drop here, autocomplete there, and here we go, an ERP system! In reality though...
Saqib Shaikh lives in London. The UK doesn't have H1Bs. His LinkedIn suggests that he went to high school in England. He has (what sounds to me like) an English accent. Put some effort into the lazy racist joke at least.
In my first developer job, I worked with a blind developer. Back then it was green screens, ksh, vi, and a screen reader. That was over twenty years ago.
I'm surprised the process is very much the same today as it was then.
In my experience it's pretty much impossible to get any kind of UI done in Xcode, at least as of Xcode 7. I'm a totally blind programmer who has a Macbook Air that I run windows on most of the time. I'm not sure if my issues with Xcode were do to a lack of experience with Voiceover on the Mac, lack of knowledge about iOS programming, or lack of accessibility in Xcode. It would be nice if Apple could come out with a basic guide targeted at Voiceover users to create a simple app that allowed you to enter text into a text box, click a couple of buttons and have something happen, etc. I've had a really ahrd time following any guides online to get started with iOS programming. Not sure what other blind developers experiences are though.
This is great. I always wondered how blind developers managed to navigate IDEs at a decent pace.
I can't help but wonder how Saqib Shaikh navigates through log files for large applications though. Log files usually generate a large amount of noise in the form of time stamps, thread names, log levels, stack traces, etc. that overtime I've learned to "scan" instead of read to find what I am looking for. I would love to see a video of him navigating through a log file to see how he does it. Maybe just lots of string searches?
1 - does sun light have no physiological/psychological value for blind people? the sun light might still be captured on the skin or even in the eyes and trigger relevant biological processes.
2 - having a monitor is also important for a blind person when working in a team.
I'm totally blind and have what I call no usable vision. I can see light if it's bright or the sun is shining. I can't tell anything about the environment from my vision though other then if there is bright light or not. Anecdotally from talking to other blind people the ability to see light helps set your body clock. Although I keep a fairly normally sleep schedule I know blind people who have no light perception and have a lot harder time keeping a normal schedule with out medicine.
The vast majority of visually impaired and legally blind have some degree of sight or at least light perception. The tiny percentage that does not still needs/like sunlight for vitamin D etc, but you don't get much sunlight form a window.
As a blind programmer who uses a laptop I still got a second monitor that was not currently being used by anyone. My co-workers thank me when they don't have to hunch over to look at my laptop screen when going over code.
Quite a few legally blind people have some remaining vision that is both useless for them and extremely annoying, so they wear sunglasses, and prefer not looking towards light sources.
Sounded and behaved like the default narrator that comes with Windows. They all tend to rely on the same APIs, though, so what they “see” of the screen or the focused control is very similar.
Listening to the beginning of that video as a blind programmer I'm 99% sure he's using Jaws for Windows.
http://www.freedomscientific.com/Products/Blindness/JAWS
I program in Java and Eclipse for my day job but have found Visual Studio to be really nice to use and accessible with Jaws.
66 comments
[ 3.1 ms ] story [ 120 ms ] threadThat seems very common for systematic text-to-speech users, I saw a video a few months back of a lawyer (?) demonstrating ios's accessibility (rotor, voiceover, blind curtain, …) and the TTS speech was so fast it sounded like white noise to me.
https://finance.yahoo.com/news/david-pogue-on-iphone-voiceov...
The "visual cortex" area is not deactivated but gets re-tasked with non-visual processing tasks, even in "late blind" people (people who lost their sight after age 12): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3667661/
Considering the amount of visual information the sighted process more or less effortlessly, that's a pretty significant amount of processing capabilities which can get retasked with e.g. auditory processing.
I'm still impressed at how he found the syntax error, though.
Edit: wrote semi colon instead of " double quote
"Jarvis import library foo" - could be said from anywhere in file
"Jarvis jump to file bar.c"
"Jarvis declare var current temperature equals zero" - var currentTemperature = 0
"Jarvis write for enumeration on var word over var name list"
"Jarvis build and run" or simply "Jarvis run"
By the way, I know many people don't like to talk to a computer, can't imagine how it could be faster than typing, etc. However, I believe that once an assistant is done right, more than a few people would be converts.
You may be interested in the guy that did all his coding with voice for a while because he couldn't use his hands, and still uses it to this day about half of the time. https://www.youtube.com/watch?v=8SkdfdXWYaI
https://github.com/melling/ErgonomicNotes/blob/master/progra...
No one said that you have to do everything. Start with what's possible then iterate.
(btw, if anyone knows anything about software/hardware to do that? let me know)
It would good to have the options to voice or gesture command a desktop computer.
I created speech bindings for my most common navigation tasks:
- "Down" -> presses PgDn
- "Down a bit" -> presses "down" a few times
- "Switch" -> presses Cmd+Tab (next application)
- "Backtick" -> presses Cmd+` (next window)
- "Apple L" -> presses Cmd+L (focus address bar)
Latency was good enough. Bindings have a higher priority than dictation, so if it hears one of your custom bindings: it will immediately action it instead of waiting to hear a full sentence.
Mac version was mostly only reliable for triggering keyboard shortcuts. Dictation of sentences worked pretty well, but there were (many) significant bugs that meant it was not generally viable.
I eventually stopped because of one of the following (can't recall):
- OS updates broke it - lost the license in a computer switch - hands got better
Almost the identical use case - I sometimes sit back and use gestures to scroll, swipe browser back, swipe between desktops.
An external trackpad would also do the trick, but the mouse is ergonomically nice for when you're stretching or leaning back.
Oh, and it must work locally instead of sending everything to the cloud.
For now, it would be helpful to many if there were no such limitations. In other words, comments like "Make it even smarter and it can't go to the cloud then I might use it" can safely be ignored for now. There's a market for an MVP that does a lot less.
Or my wife ready to kill me as I'm saying code while she's watching tv. :)
And, IMO, even mumbling would be better than the current open office constant flow of loud conversations (one of which I'm trying to drown out with headphones this very moment).
https://www.youtube.com/watch?v=8SkdfdXWYaI
In the same vein, I am learning morse code. Any resources helping with this would be appreciated.
I would like to use morse code for silent, no-look I/O on my phone. I'm not sure vibrations can be quiet enough; might have to use something special-purpose plugged into the headphone jack.
Make the whole keyboard area a blank "tap zone" where you can just use any digit to tap text into an input field.
https://github.com/gitonwithit/MorseKeyboard
https://github.com/whymarrh/morse-code-keyboard
https://play.google.com/store/apps/details?id=com.Input.Mors...
"Please stop shouting at the whiteboard. It doesn't work that way. I need you to pick up one of the dry-erase markers and write your algorithm to invert a binary tree."
"What? Who codes with their hands? I need my assistant!"
But seeing "Visual Studio" and "blind" together did make me look twice.
Listen Mr. Senior Principal Architect, not only cheap H1B Indian guy, but even cheap blind H1B Indian guy can replace you!
What I don't like about MS is that in their promo materials developing software is so easy. Drag and drop here, autocomplete there, and here we go, an ERP system! In reality though...
I'm surprised the process is very much the same today as it was then.
"How can you program if you're blind?" | http://stackoverflow.com/a/453758
I can't help but wonder how Saqib Shaikh navigates through log files for large applications though. Log files usually generate a large amount of noise in the form of time stamps, thread names, log levels, stack traces, etc. that overtime I've learned to "scan" instead of read to find what I am looking for. I would love to see a video of him navigating through a log file to see how he does it. Maybe just lots of string searches?
What was odd was that:
1 - they were given the worst place in the room: no windows, literally facing a wall. Took some time to realize why they didn't complain.
2 - The computer had monitors. Then I learnt that Windows (don't know what version it was) wouldn't work it no monitor was attached.
1 - does sun light have no physiological/psychological value for blind people? the sun light might still be captured on the skin or even in the eyes and trigger relevant biological processes.
2 - having a monitor is also important for a blind person when working in a team.
So, this may have been by choice.
I know blind people wouldn't need one, but somehow the lack of it brings up the wow factor.
you could even ask "which lines have if statements?"
for something like this wouldn't you be talking to an AST rather than a text editor?