I love the eye tracking. Even without having any issues with my hands, I'd like to have quick mouse movements without having to take a hand off of the key board. Zooming in to have more precision is clever.
It seems the current generation systems aren't working for Linux yet, but that gives me hope to have a way to work with it at some point in the future.
I think parent is talking about the Tobii. I was also very very excited about it (I can’t find a mouse or trackball that won’t give me hand or shoulder or wrist pain) but then I saw it only works on windows... and nothing else. No Linux support. So that’s a no go.
If someone knows of such a device for Linux, I’d love to hear about it!
Would you mind elaborating on this? I'm ready to buy a Tobii but would like to know what the model 5 caveats are and how long you think it'll be WIP. And if you think the improved specs on the 5 make it worth waiting for. Thanks!
Keeping the hands on the keyboard would really increase productivity and reduce repetitive motion injuries. I know that editors like vim and emacs let you keep your hands on the keyboard, but interacting with anything else requires the mouse/touchpad.
At least 10 years ago, I imagined that the mouse could be replaced with something like a brainwave reader. You can train your brain to emit simple patterns that can be picked up either with a headband or remotely. It would take individual training, but with a few basic patterns, one should be able to do mouse movement and clicks. The eye tracking seems like the next best thing, especially since it is already implemented.
As a developer who also has cubital tunnel syndrome, having keyboards that places commonly use modifier keys [0] on your thumbs, such as Kinesis advantage or Ergodox, helps reducing the fatigue and symptoms immensely.
[0] it is called Emacs pinkies for a reason, and yes, my CTS is totally emacs’s fault as well.
"""In the mid 90s I had bad hand pain, so bad that most of the day I could only type with one finger. The FSF hired typists for me part of the day, and part of the day I tolerated the pain. After a few years I found out that this was due to the hard keys of my keyboard. I switched to a keyboard with lighter key pressure and the problem mostly went away.
My problem was not carpal tunnel syndrome: I avoid that by keeping my wrists pretty straight as I type. There are several kinds of hand injuries that can be caused by repetitive stress; don't assume you have the one you heard of."""
Stenotype keyboards are way better from a RSI point of view, though. Unfortunately, no one has come up with the "theories" or systems that would allow use of them for specialized input like programming.
I've seen a few conference presentations about using steno for programming. It does seem to involve building your own library of commands, but people seem to have done it and use it daily.
I have a planck keyboard, but I think it's small size isn't fantastic for hand position. I think I may eventually purchase an ergodox ez to get better hand positions.
Am I the only one who just keeps their hands/wrists straight on a regular unsplit keyboard? I've always naturally done this and never had any problems with strain.
The best way to describe it is that the home keys aren't ASDF, but more like WEFV with my forearm angled so it's straight from elbow to fingertip. This exaggerates the effect but demonstrates it clearly; the real home positions are more like intermediate points between keys.
Yeah, same. I find keeping my fingers on the home keys makes me twist my wrists outwards, which gets uncomfortable pretty fast. I mostly try to minimise wrist action in general.
I've actually had a Planck EZ for a few weeks now and I agree it's probably not very ergonomic for the classic qwerty touch typist, or at least not much better than a normal keyboard barring the programmability.
But since I don't do that anyway I find the keyboard to be pretty nice in terms of customisability and avoiding stretching.
In general I feel my hands are used most naturally in close proximity to each other (at roughly abdomen height) so I'm drawn to small keyboards with lots of modifier keys. A spherical keyboard would be pretty interesting to try out.
On my keyboard - which is.. I don't want to say 'full size', because maybe it's '80%' or whatever, but it's a regular external keyboard with numpad - f & j are centre to centre 60mm apart.
If I outstretch my arms, straight, my index fingers are pretty much in line with my shoulders.
So no, when I use my regular unsplit keyboard my hands/wrists are not straight, because my shoulders are not 60mm apart..? Am I missing something?
I believe GP was talking about elbow-hand being completely straight. That would make sense if their elbows are on arm rests or some such thing.
It does sound interesting. I did type this comment trying it, and it was pretty comfortable, with the exception of my right wrist. That being said, I've got kind of weird seating set up.
Oh.. then I'm not sure what we're doing differently, or what the difference is in hardware, but my arms/hands are straight from elbow to fingertip when I rest on 'home row' 'adsf'/'jkl;' - if I change to 'wefv' it's a significant contortion, if that's what it looks/feels like for some people using 'asdf' then I understand the problem and am not surprised they don't put up with it!
If I stick my elbow way out I can make 'wefv' as natural to me as 'asdf'. So I suppose either my elbows are more tucked in, or my keyboard's wider ('dfgh' for example contorts the other way, and analgously to 'wefv', 'rthn', or better but less analogously 'erth', is more comfortable there).
Yes, your last paragraph is my natural position, with the elbows farther out. 'wefv' exaggerates the effect to demonstrate; the real position is about halfway between 'wefv' and 'sdfv'.
Had pain a few years ago, switched to Ergo 4000's everywhere and have never had any pain since - depending on desk height I'll use the front riser but generally don't need it.
I love them so much I have spare new-in-box ones stashed as a hedge for the day they stop selling them.
They are about the cheapest good quality ergo keyboard I could find.
Not to flame-bait, but this is actually one of the reasons I've doubled down on vim. I like that I can function perfectly fine with one finger, I even managed to write a bunch of code using vim on my phone, and was surprised how well it translated.
You could definitely set up toggled modifier keys and do the same with emacs, FWIW.
The key issue (pun intended) with normal keyboard is that you use your weakest fingers for ALL of your modifier keys. By shifting that responsibility to your thumbs you greatly reduce your little fingers’ workload (and hence chance of CTS)
That's a great idea! This would really push up input speed. I'm surprised I haven't seen any implementations in the wild yet, although research has been done to explore this [0]
In my experience (as a developer of eye gaze interfaces, not an everyday end user) it is often more efficient to have really good next-word-prediction (such as with the Presage engine) combined with single-letter dwells, rather than using "swipe-like" spelling, where you're committed to tracing out the whole word.
There are alternative virtual keyboard styles (typically made for use with a mouse and impaired motor control) which would work better. Fewer "fine" movements.
Combined with predictive typing, and you could go pretty far.
I don't know about that, but I'd love to have eye tracking for focusing windows. Since I don't generally maximize windows, I find that I'm often typing hotkeys for the window I'm looking at, not the window my mouse is on (or was last focused, depending on your wm)
There are some high tech breakthrough in this space. You can have microphones that hear the faintest of internal voices and even non-invasive ways to hook on the nerves to get your internal monologue.
Exploratory building of rich non-traditional (not necessarily handless) user interfaces is becoming increasingly accessible. For instance, here's a web demo of Google's MediaPipe's face and iris tracking[1]. And hand tracking[2] with a downward-facing camera, permits enriching keyboard and touchpad/tablets: annotating events with which finger was used, and where on a keycap it was pressed; hand and finger gestures in 2D and 3D; and positional touchless events. And speech to text... sigh.
But doing sensor fusion is hard. And strongly impacts system architecture. "Launch the missiles"... 1000 ms later... oh, nope, that was "Lunch is mussels in butter". "Spacebar keypress event"... 50 ms later... "btw, that was a thumb, at the 20% position". "Ok, so 2000 ms ago, just before the foo, there was a 3D gesture bar". So app state needs to easily roll backward and forward, because you won't fully know what happened now until seconds from now. Upside is traditional "have to wait a bit until we know whether mumble" latencies can be optimistically speculated away.
When I started my undergraduate degree, there was a PhD student everyone knew -- he looked like he ran the Linux User Group, which he did, he coded his own window manager, wrote games in Haskell etc.
At some point, he got some hand injury that meant he couldn't type normally for months. He would only have been 21 or 22.
This scared me, so since then I've followed the standard advice on avoiding computer-related injuries very carefully. I use a desktop computer with an external, adjustable monitor. I use ergonomic keyboards, and use a mouse with either hand. I learned to touch-type with the Dvorak layout, I have an adjustable desk, etc.
I haven't had any problems, and I don't know which (if any) of these actions helped, but I'm surprised when I see other developers apparently happy to hunch over a laptop keyboard for 8 hours, while sitting on an awkward chair in a coffee shop. Why do this to yourself?
I think ..... most people have no idea what they actually look like when hunched over a keyboard. It would do wonders to have a large mirror next to our workspace so one could see just how preposterous they're contorting their body.
I had a bit of a scare around the same time. I was following dual passions of computer science and piano playing, meaning around 12 hours a day of typing and/or piano-ing. I started to develop regular tendinitis in my wrists and fell down a rabbit hole of ergonomics and posture. With computers you can replace keyboards if you have money, but I was a musician in college. Also, you can't really change a piano keyboard all that much (and you really wouldn't want to) so I did a real deep dive into posture and getting the most out of what I already had
A few years back I started buying disability insurance, in addition to paying more attention to my ergonomics.
My greatest economic asset is the years left in my career. For ~$1500/year I insure that. It's not cheap, but it would pay out until I was 65 at some reasonable percentage of my normal salary in the case I was severely injured, or otherwise unable to work. Anywhere from an ergonomics caused hand/arm injury, a bike-riding caused head injury, a wood-shop caused eye damage, etc.
Consider grabbing it while you're young and working in a lucrative career.
Assuming physical injury (like RSI or trauma), yes, it's getting easier and easier.
But there are many other ways you could be injured and unable to work. My straw-man injury when I bought the policy was: "I'm a new skier, which is famous for major head trauma". Similarly, cancer, which directly or via treatment, causes exhaustion is another example of disability that is not solved with "if only I could just input text easier".
I'm glad there's so much work in accessibility going on, and that hardware, machine learning, and social awareness has gotten widespread enough to let non-huge companies dive in and make real progress. It's important to open the field to as many people who we can.
I just want to be sure people don't rely on technology being advanced enough at the exact time you may need it. You can easily get unlucky, and insurance exists to pay somebody else to take the risk.
I have a similar story. When I was in college I got a repetitive stress injury from practicing piano too much. I ended up becoming a programmer but I have followed ergonomic practices religiously in the 40 years since then. My kids make affectionate fun of my perfect posture. But I have worked with the computer keyboard for going on five decades and have never come close to suffering a repetitive stress injury.
I have had an issue with dictation software... I want a dictation software to read me what it typed on screen. Right now the software expects you to check what it wrote and you have to necessarily keep looking at the screen.
Call me lazy but I wish for a method where I could lay back, speak a sentence, wait for transcribing, listen to what was written and ahead I go.
when I was writing my last novel draft using nuance dragon 15, it had the 'read that' command which would read the last phrase I inputted, but that wasnt enough since I had a wireless headset that let me wander the house talking to the computer, but still needed to see the screen to know if it got the dictation correctly.
I ended up fishing out a cheap $120 portable walmart windows laptop 2 in 1 and VNCing it into the desktop so i could see the screen, carrying that around with me worked.
my point exactly. all dictation softwares expect you to keep looking at the screen. your "I had a wireless headset that let me wander the house talking to the computer" is what i want a dictation software to do. sadly like you i am left unsatisfied
I wonder if the author has considered integrating his voice interface with an AI resource like GPT-3. I could see future development being done in this fashion.
I personally have had carpal tunnel release surgery and bilateral rib resection for thoracic outlet syndrome. These surgeries fixed most of my issues but for the last couple of years I haven't tolerated sitting well because of my back so I use a standing desk. I've thought about a computer interface that is the opposite of this article. Something that requires more non-repetitive physical movement so that you're exercising while you program. Sure it would take longer to type but we all know working on a computer all day is bad for our health.
Not sure if this helps but I had a similar issue with my joints (arm, shoulder, wrist) where I would scream in pain when lifting a small book or anything above 3 lbs.
The issue was caused by my immune system acting up which would cause my tendons to become inflamed. After a year of misdiagnosis, Doctors found some heavy medication (methotrexate balanced with plaquenil) to help regulate it. One of the medicines side effects was listed as ‘death’ on the label.
After a year of experimenting, I found that major diet changes (lots more hearty greens, way less sugar and carbs, no caffeine or alcohol), improved sleep, reduced stress (quit my stressful job) completely alleviated my symptoms. I would still have flare ups from time to time which I reduced via physical therapy / exercise (to strengthen muscles supporting my tendons).
Just sharing as I had a somewhat similar condition and was surprised that the fix didn’t have to be a pill.
Drastically reducing “added” sugars (which includes alcohol), whether end-user added or added as part of the industrial food “manufacturing” process does wonders for one’s health.
Can confirm. Cut out almost all sugar from my diet. I've seen dramatic improvements in my health with the most significant being the elimination of brain fog.
I was misdiagnosed inflamed tendons and it turned out to be something else, hah
HN Reader take this message: medicine can be a to and fro, go back if you're not convinced and get a second opinion if you think somethings still wrong :)
It sounds like I was in the same boat as you but I had multiple issues with my left leg.
Saw multiple specialists and had been prescribed multiple different medications all with the same result of nothing but with new issues thanks to some side effects.
One day I decided I wanted to lose some weight and cut down my carb intake down to a very low amount per day. Shortly after making this change to my diet I noticed the problems that I was experiencing with my leg slow started to go away.
I wouldn't say the problems with my legs have completely gone away due to my diet change but it has improved my quality of life significantly.
I had pain and weakness in my right wrist for years, and also had many rounds of misdiagnosis.
It turned out to be a benign tumour. Surgery was my answer, but it's only a partial one. It's a soft tissue tumour, the surgeon didn't expect to get all of it, and it's growing back now. It's been about seven years, and I expect I'll need surgery again within seven more years to be able to do simple tasks like lift objects with that hand still.
Sometimes it's not diet - but it's like the old climate change joke. What if I switched to a better diet and a healthier lifestyle, for no good reason?
Hemangioma in my case. I had an MRI but in the end it was a surgeon with the right experience looking at an earlier ultrasound that diagnosed it.
I had trouble believing he'd diagnosed it when he glanced at the ultrasound and calmly said what it was. Given I'd had medical practitioners send me for neurological studies to check it wasn't all in my head, suggest fusing my wrist to stop the pain, tell me there was nothing visible in X-ray, MRI, or the same ultrasound report the surgeon diagnosed me from... It seemed unlikely anyone could do an apparently simple diagnosis like that.
> The biggest issue I've found so far is voice strain
My singing teacher started out as a voice therapist.
He is still working with a lot of people with damaged voices that are there for therapeutic reasons as well as professional opera singers that there to level-up their game.
He says, the best thing you can do for your voice is to sing.
Singing will give you the proper "support" from your lungs and train the vocal muscles without wearing them out.
One of the key aspects here is producing high quality vowels, and don't let the sound be disturbed by consonants.
This is opposite to how the voice is used when speaking (in most languages), where consonants are of primary importance for people to understand what you are saying.
However, consonants are interruptions in the air-flow and can easily wear the voice out and cause stain.
I've been wanting something like Talon for years. Easily customized voice recognition for commands (you hear me say 'blah', you type/start a program/click a button labelled 'blah') is something I've not had since I was very young and tried to trick out LCARS x32 using only Windows XPs speech recognition software using the crappy stick mics that lived on top of every single beige CRT monitor at school.
I don't know why, but I've often thought about the situation the Josh finds himself in. IE - having to find a different way of building software if I couldn't use a keyboard and mouse due to medical condition.
The solution I always imagine is paying someone or having my employer pay someone to strong style pair program with me. Perhaps a student, junior developer or even someone unfamiliar with software development entirely.
For those unfamiliar with strong style, this rule sums it up: "For an idea to go from your head into the computer it MUST go through someone else's hands". Like the standard driver / navigator pair programming technique but with the navigator never touching the keyboard.
In the case of someone completely unfamiliar with software development I imagine that there would initially be a dramatic high / low skill gradient between us, with the person essentially transcribing. However given the intensity of the practise I think this gradient would level out quite quickly.
> The solution I always imagine is paying someone or having my employer pay someone to strong style pair program with me
Wow, that's some impressive out-of-the-box thinking!
The OP's situation is something I worry about, as I'm worried my neuropathy will reach a point where I can't use my arms to type/mouse all day. I spent some time researching speech input for coding a year ago, and found a few fledgling solutions, but I'd need to expend a lot of time to get something remotely workable.
Your idea is, if I may say so, genius, and having that as a backup sets my mind greatly at ease, so thanks.
I'm sorry to hear about your neuropathy, I'm glad my suggestion eases your mind!
On pairing in general: For a long period of time when I worked at bigger companies (before co-founding my current venture) I pair programmed 90% about of the time.
For the most part I really enjoyed it, we alternated pairs quite frequently and I worked with people at a range of seniorities. I really enjoyed alternating between playing the role of teacher vs learning from someone much better at the (insert programming language, database, business domain).
I think during that time I also had some of the most stimulating and deep conversations about software development. When pairing, pairs have a deep shared context that's hard to replicate in other scenarios.
There are some drawbacks though, the main one being it's extremely full on and emotionally tiring. I'd class myself as a high functioning introvert and after a day of pairing I needed some serious regenerative quiet time!
My advice to someone considering this suggestion would be to try some pairing now with friends or colleagues and get a feel for it and see if it works for your personality and working style.
> I'd class myself as a high functioning introvert
Hah, I'm stealing that phrase to describe myself too!
I've never "formally" done pair programming, but have sat and coded/reviewed with colleagues for short stints, and generally found it OK.
From time to time I have to do full days (or several days) of workshops, and find those really tiring and emotionally draining. I always think I fare better 1-1 with people though, so if pairing was a necessity, I'm sure I could adapt.
I touch on this in the article, but I should say: I'm an edge-case when it comes to CTS. Most cases resolve spontaneously. Part of it is that my nerve dislocates when I bend my elbow, and this mobility causes a bunch of additional friction / inflammation. This happens naturally to about 13% of the population, and mine is particularly pronounced.
All this to say, if you ever do start to get a burning or tingling in your elbows or wrists, or get numbness in the hands, it'll likely go away on its own, or with conservative treatment (a physical therapist is a great first step!).
And, for those who also fall into my unlucky percentile, hopefully it helps to know that there are tools and a community that exist to let you keep working without needing to use your hands at all!
Ah, that sounds like what I have! Subluxation of the ulnar nerve is what they told me it was - I could feel the nerve popping over my medial epicondyle. We tried steroid shots to reduce inflammation, but it came back after about nine months. It got so bad that I couldn't fully extend my arm or bring it in past ~45 degrees from extended. The inflammation showed up on an MRI. I had surgery to relocate the nerve to the inside of my elbow and have largely recovered from it, though it does flare up every so often.
The scare got me very interested in accessibility and I've looked into using Talon with eye tracking and stenography via Plover to improve my ergonomics, but I've never fully committed to any of them, but I'd love to know what communities exist. A Kinesis Advantage and a vertical mouse seem to keep it at bay on all but the longest of work days.
> I've heard that learning Vim can make this much more effective.
This doesn’t surprise me; Talon is a lot like Vim.
They’re both modal: Talon’s command mode roughly matches Vim’s normal mode, and its dictation mode roughly matches Vim’s insert mode.
Talon’s ordinals apply to Vim operations (normal mode) as well: operations can be preceded by a number which normally says how many times to do it. Talon’s “go left ninth” would be 9h or 9← in Vim. (h goes left one character.) “One zero third” is more cumbersome because in Vim it entails switching between modes (Talon and Vim modes don’t match precisely), but starting in normal mode, a1<Esc>3a0<Esc> works to append 1000 where the cursor is. (a to enter insert mode after the current position, 1 to type a 1, Escape to return to normal mode, 3 to say “do the next thing three times”, the next thing being appending a zero.)
Both Talon and Vim value this sort of consistency in being able to build bigger things from smaller pieces.
So I imagine Talon would work quite nicely with Vim, once you get talon to shift out of the way in some cases and let Vim handle those things. (It’s similar with browser extensions that let you interact with the browser Vim-style: when the page in consideration has extensive key bindings, especially Vim-like things like j/k for down/up, the extension actually gets in the way.) I expect you’d find that having used one made learning the other a good deal easier, since they’re working in just the same ways.
Thanks for this! Sorry it had to come about as a result of injury/disease. Hopefully you're not going to hurt your vocal tract -- are there precautions you take against this?
My interest currently is in voice-assistants and developing something like a personal assistant / code-monkey / grad-student.
I did have a stroke a few years ago, from which I've recovered, but it made me realize that almost all of my work, hobbies, aspirations, are tied to fairly extensive hand-eye integration. It's good to start building a Plan B before I need it.
On a(nother?) side-note, my 88 year old dad still codes and his experiences have really shown me how bad we are at accessibility, as a developer community.
One final thing -- watching you work in this way highlights for me how low-level a lot of the stuff we do as developer is. There's a certain amount of just text massaging that seems irrelevant to what we're actually trying to get done (although it can be fun/soothing/aesthetic).
EDIT> The last point is why I can't give a fark about Vim/Emacs/whatever debates and obsessing. It seems like it might be fertile ground, but IMNSHO it's a trap.
I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enough better to make it worthwhile.
There are moments where I think this is going to be the future of programming, since code as text is only the easiest for as long as typing is the easiest way to record things unambiguously.
But for the most part it’s still pretty frustrating. If ASR systems can get the sentence error rate down by an order of magnitude or two, I sincerely think this will take off not just for accessibility, but for normal use.
Until then, it’s a PITA that is saving my career nonetheless.
Talon actually can work with Dragon; it started out requiring Dragon but has since grown its own voice recognition. Anecdotally, a lot of folks on the talon slack seem to be refugees from Dragon-based solutions, and a common observation is that Talon's voice recognition is much lower-latency than Dragon. Unfortunately I can't confirm this as I haven't used Dragon myself.
My impression of talon's speech recognition is that it's good enough for voice coding, but could be improved when it comes to dictating English prose. That said, there are promising avenues of improvement. If you pay for the Talon beta, there's a more advanced voice recognition engine available that's much better at English, and you can also hook into Chrome's voice recognition for dictation.
More specifically, "can work" means: if you run Talon on Windows or Mac alongside Dragon, and don't configure Talon to use its own engine, Talon will automatically use Dragon as the recognition engine.
Unfortunately it looks like Dragon on Mac isn't a thing anymore, which is a shame because my main dev machine is a Mac. Or at least it used to be, before I needed to start dictating everything.
Talon can also use Dragon remotely from a Windows machine or VM, kind of like aenea but more tightly integrated (the host is driving instead of the guest).
And I do have some users voluntarily switching from Mac Dragon to the improved speech engine in the Talon beta. Mac Dragon has been kind of buggy for the last few years so you're not missing much.
Any chance you have pointers on how to set that up? You'd probably laugh/cry to see my setup right now, with my Windows desktop on the left monitors and my MacBook on the right monitors because I need both... purely because Dragon is only sold on Windows since this started been an issue for me. A more tightly coupled super-aenea sounds pretty fantastic.
Sure: First run Talon on both sides. Then go to your talon home directory (click talon icon in tray -> scripting -> open ~/talon). There's a draconity.toml file there.
On the Dragon side, you need to uncomment and update the `[[socket]]` section to listen on an accessible IP.
On the client side (Mac in your case), uncomment / update the `[[remote]]` section to point at your other machine.
You also need to make sure both configs have the same secret value.
From there, restart Dragon and look in the Talon log on the Mac side for a line like "activating speech engine: Dragon".
To prevent command conflicts, I recommend setting Dragon to "command mode" (if you have DPI), and only adding scripts to Talon on the Mac side.
If it doesn't work, you can uncomment the `logfile` line in draconity.toml on the Dragon side, restart Dragon, and look in the log to see what's going on.
Do you know of any workflows like this using entirely open source software?
EDIT: Seems like caster itself has instructions for an open source recognition engine to pair with it. Not sure how accurate it'll be but I'm going to give it a shot!
I hate the idea of relying on closed source software to be able to continue in my profession. If this works I'm definitely going to be donating to the FOSS options
I do not use the wav2letter@anywhere inference frontend - I trained the acoustic model using the facebook upstream code, but the decoder is almost entirely new, and on Windows I use Pytorch for inference.
Talon ships with a libw2l.so/dylib on Linux/Mac built from my open source repos.
Feels like wav2letter will not be actively developed anymore. Understandable since it is hard to compete with Pytorch with a custom NN toolkit. Any plans to move to Pytorch/Tensorflow?
I don’t think that’s right at all. They moved development to the Flashlight repo. It seems very actively developed to me. Last commit 3 hours ago. wav2vec 2.0 blog post also went up last month (September) and the current state of the art iirc is a Google model based on Facebook’s Wav2vec 2.0 work.
For my own use I’ve already built Pytorch and CoreML frontends, with a shared model format (can convert models to/from wav2letter format and my custom format), and I have the ability to create new models in these frameworks from wav2letter architecture files.
I still run my training in the wav2letter framework, but for compatible training in Pytorch I would mostly just need criterion implementations. I assume warpCTC is fine for the CTC models. There’s also a third party Pytorch ASG criterion package but I haven’t tried it yet.
Yeah there is the Kaldi backend for caster, which I've tried on my Mac machine, since Dragon isn't a thing on Mac. Unfortunately it's not nearly as good :-(
I'd like to try to record my usage of Dragon that I could fine-tune my own model, but it's harder to get around to hobby coding projects like that now that coding is more of a pain in the ass.
The unfortunate reality in my case is that using Dragon is the least frustrating way to keep working. I don't think closed paid ASR models will stop being noticeably better better than three open ones until the state-of-the-art error rate is basically zero.
I do like that caster is open source where talon isn't, but preferences like that are pretty low priority when say your career is on the line.
> I do like that caster is open source where talon isn't, but preferences like that are pretty low priority when say your career is on the line.
For sure if its on the line right now you use what is available and works best right now, but the reality of niche software is the companies/people backing it tend to close down or move on as they realize the market is too small to sustain them. Or Microsoft updates its OS in a way that forces them to expend considerable time making it functional again and they just don't have the bandwidth for it so you're stuck for some amount of time or indefinitely.
If you have no other choice, its definitely better than nothing and you want to use what enables you to do your job the best. But in terms of where I'd be willing to throw my financial support? It'd need to be something for the better of the coding community as whole. For me, this means an open source tool or set of tools, not a proprietary system.
For RSI folks, my experience is that RSI can “move” to other parts of the body pretty easily, including eyes and vocal chords. Somehow doing “work” with a part of the body can strain it, even when talking all day long socially wouldn’t.
140 comments
[ 4.6 ms ] story [ 2856 ms ] threadIt seems the current generation systems aren't working for Linux yet, but that gives me hope to have a way to work with it at some point in the future.
This isn’t correct. Talon supports eye tracking on all of Windows, Linux, and Mac with the same hardware.
If someone knows of such a device for Linux, I’d love to hear about it!
* The Tobii 5 does work, but is a work in progress and has some caveats until I do more work on it.
At least 10 years ago, I imagined that the mouse could be replaced with something like a brainwave reader. You can train your brain to emit simple patterns that can be picked up either with a headband or remotely. It would take individual training, but with a few basic patterns, one should be able to do mouse movement and clicks. The eye tracking seems like the next best thing, especially since it is already implemented.
[0] it is called Emacs pinkies for a reason, and yes, my CTS is totally emacs’s fault as well.
Edit: Not carpal tunnel syndrome, as he explains in https://stallman.org/stallman-computing.html
"""In the mid 90s I had bad hand pain, so bad that most of the day I could only type with one finger. The FSF hired typists for me part of the day, and part of the day I tolerated the pain. After a few years I found out that this was due to the hard keys of my keyboard. I switched to a keyboard with lighter key pressure and the problem mostly went away.
My problem was not carpal tunnel syndrome: I avoid that by keeping my wrists pretty straight as I type. There are several kinds of hand injuries that can be caused by repetitive stress; don't assume you have the one you heard of."""
http://www.openstenoproject.org/plover/
The Planck keyboard is cramped. It's worse than a cheap Microsoft ergonomic keyboard.
Ergodox, Diverge, Keyboard.io, really any split key keyboard does worlds for keeping hands straight and are fully programmable to very high degrees.
The best way to describe it is that the home keys aren't ASDF, but more like WEFV with my forearm angled so it's straight from elbow to fingertip. This exaggerates the effect but demonstrates it clearly; the real home positions are more like intermediate points between keys.
I've actually had a Planck EZ for a few weeks now and I agree it's probably not very ergonomic for the classic qwerty touch typist, or at least not much better than a normal keyboard barring the programmability.
But since I don't do that anyway I find the keyboard to be pretty nice in terms of customisability and avoiding stretching.
In general I feel my hands are used most naturally in close proximity to each other (at roughly abdomen height) so I'm drawn to small keyboards with lots of modifier keys. A spherical keyboard would be pretty interesting to try out.
If I outstretch my arms, straight, my index fingers are pretty much in line with my shoulders.
So no, when I use my regular unsplit keyboard my hands/wrists are not straight, because my shoulders are not 60mm apart..? Am I missing something?
It does sound interesting. I did type this comment trying it, and it was pretty comfortable, with the exception of my right wrist. That being said, I've got kind of weird seating set up.
If I stick my elbow way out I can make 'wefv' as natural to me as 'asdf'. So I suppose either my elbows are more tucked in, or my keyboard's wider ('dfgh' for example contorts the other way, and analgously to 'wefv', 'rthn', or better but less analogously 'erth', is more comfortable there).
I love them so much I have spare new-in-box ones stashed as a hedge for the day they stop selling them.
They are about the cheapest good quality ergo keyboard I could find.
You could definitely set up toggled modifier keys and do the same with emacs, FWIW.
Very cool stuff, thanks for sharing
[0] https://pdfs.semanticscholar.org/1170/44feb247a82ab81a30b57f...
Long before that, Optikey has had "multi-key selection" which works in the same way, see https://youtu.be/HLkyORh7vKk?t=10
In my experience (as a developer of eye gaze interfaces, not an everyday end user) it is often more efficient to have really good next-word-prediction (such as with the Presage engine) combined with single-letter dwells, rather than using "swipe-like" spelling, where you're committed to tracing out the whole word.
Combined with predictive typing, and you could go pretty far.
But a whisper mode might prevent ear strain for those working around you :)
https://www.smithsonianmag.com/innovation/device-can-hear-vo...
But doing sensor fusion is hard. And strongly impacts system architecture. "Launch the missiles"... 1000 ms later... oh, nope, that was "Lunch is mussels in butter". "Spacebar keypress event"... 50 ms later... "btw, that was a thumb, at the 20% position". "Ok, so 2000 ms ago, just before the foo, there was a 3D gesture bar". So app state needs to easily roll backward and forward, because you won't fully know what happened now until seconds from now. Upside is traditional "have to wait a bit until we know whether mumble" latencies can be optimistically speculated away.
[1] https://viz.mediapipe.dev/demo/iris_tracking ("run" button is top right) [2] https://viz.mediapipe.dev/demo/hand_tracking
At some point, he got some hand injury that meant he couldn't type normally for months. He would only have been 21 or 22.
This scared me, so since then I've followed the standard advice on avoiding computer-related injuries very carefully. I use a desktop computer with an external, adjustable monitor. I use ergonomic keyboards, and use a mouse with either hand. I learned to touch-type with the Dvorak layout, I have an adjustable desk, etc.
I haven't had any problems, and I don't know which (if any) of these actions helped, but I'm surprised when I see other developers apparently happy to hunch over a laptop keyboard for 8 hours, while sitting on an awkward chair in a coffee shop. Why do this to yourself?
My greatest economic asset is the years left in my career. For ~$1500/year I insure that. It's not cheap, but it would pay out until I was 65 at some reasonable percentage of my normal salary in the case I was severely injured, or otherwise unable to work. Anywhere from an ergonomics caused hand/arm injury, a bike-riding caused head injury, a wood-shop caused eye damage, etc.
Consider grabbing it while you're young and working in a lucrative career.
Gestures, voice, eye tracking, ...
Being a software developer could be a job for those who physically can’t do other jobs.
Subtle input gestures with Soli would be interesting.
https://atap.google.com/soli/
But there are many other ways you could be injured and unable to work. My straw-man injury when I bought the policy was: "I'm a new skier, which is famous for major head trauma". Similarly, cancer, which directly or via treatment, causes exhaustion is another example of disability that is not solved with "if only I could just input text easier".
I'm glad there's so much work in accessibility going on, and that hardware, machine learning, and social awareness has gotten widespread enough to let non-huge companies dive in and make real progress. It's important to open the field to as many people who we can.
I just want to be sure people don't rely on technology being advanced enough at the exact time you may need it. You can easily get unlucky, and insurance exists to pay somebody else to take the risk.
It’s inefficient for everyone to do this and many cannot.
The premiums could jump 50% in a decade, for example. I’ve known situations somewhat like yours where premiums went up a lot.
Call me lazy but I wish for a method where I could lay back, speak a sentence, wait for transcribing, listen to what was written and ahead I go.
I ended up fishing out a cheap $120 portable walmart windows laptop 2 in 1 and VNCing it into the desktop so i could see the screen, carrying that around with me worked.
https://mobile.twitter.com/edleonklinger/status/128425141917...
The issue was caused by my immune system acting up which would cause my tendons to become inflamed. After a year of misdiagnosis, Doctors found some heavy medication (methotrexate balanced with plaquenil) to help regulate it. One of the medicines side effects was listed as ‘death’ on the label.
After a year of experimenting, I found that major diet changes (lots more hearty greens, way less sugar and carbs, no caffeine or alcohol), improved sleep, reduced stress (quit my stressful job) completely alleviated my symptoms. I would still have flare ups from time to time which I reduced via physical therapy / exercise (to strengthen muscles supporting my tendons).
Just sharing as I had a somewhat similar condition and was surprised that the fix didn’t have to be a pill.
Reducing sugar intake reduces inflammation (https://www.health.harvard.edu/heart-health/the-sweet-danger...).
HN Reader take this message: medicine can be a to and fro, go back if you're not convinced and get a second opinion if you think somethings still wrong :)
Saw multiple specialists and had been prescribed multiple different medications all with the same result of nothing but with new issues thanks to some side effects.
One day I decided I wanted to lose some weight and cut down my carb intake down to a very low amount per day. Shortly after making this change to my diet I noticed the problems that I was experiencing with my leg slow started to go away.
I wouldn't say the problems with my legs have completely gone away due to my diet change but it has improved my quality of life significantly.
It turned out to be a benign tumour. Surgery was my answer, but it's only a partial one. It's a soft tissue tumour, the surgeon didn't expect to get all of it, and it's growing back now. It's been about seven years, and I expect I'll need surgery again within seven more years to be able to do simple tasks like lift objects with that hand still.
Sometimes it's not diet - but it's like the old climate change joke. What if I switched to a better diet and a healthier lifestyle, for no good reason?
I had trouble believing he'd diagnosed it when he glanced at the ultrasound and calmly said what it was. Given I'd had medical practitioners send me for neurological studies to check it wasn't all in my head, suggest fusing my wrist to stop the pain, tell me there was nothing visible in X-ray, MRI, or the same ultrasound report the surgeon diagnosed me from... It seemed unlikely anyone could do an apparently simple diagnosis like that.
My singing teacher started out as a voice therapist. He is still working with a lot of people with damaged voices that are there for therapeutic reasons as well as professional opera singers that there to level-up their game.
He says, the best thing you can do for your voice is to sing.
Singing will give you the proper "support" from your lungs and train the vocal muscles without wearing them out. One of the key aspects here is producing high quality vowels, and don't let the sound be disturbed by consonants. This is opposite to how the voice is used when speaking (in most languages), where consonants are of primary importance for people to understand what you are saying. However, consonants are interruptions in the air-flow and can easily wear the voice out and cause stain.
https://serenade.ai/
He also references Tavis Rudd’s viral voice coding video, which is already 7 years old.
https://www.youtube.com/watch?v=8SkdfdXWYaI
The future is arriving slower than i would have guessed. I thought we’d be developing on the iPad using voice, gestures, and eye tracking by now.
Can I a least get “build and run” with my voice soon?
I would love to see/hear what it's like. I'd suspect adding audio to those videos would be a win on conversions.
In the meantime, here's a video that a developer using Serenade recently made about their first experience! https://www.youtube.com/watch?v=Pc-EbY1fRWk
The solution I always imagine is paying someone or having my employer pay someone to strong style pair program with me. Perhaps a student, junior developer or even someone unfamiliar with software development entirely.
For those unfamiliar with strong style, this rule sums it up: "For an idea to go from your head into the computer it MUST go through someone else's hands". Like the standard driver / navigator pair programming technique but with the navigator never touching the keyboard.
In the case of someone completely unfamiliar with software development I imagine that there would initially be a dramatic high / low skill gradient between us, with the person essentially transcribing. However given the intensity of the practise I think this gradient would level out quite quickly.
Wow, that's some impressive out-of-the-box thinking!
The OP's situation is something I worry about, as I'm worried my neuropathy will reach a point where I can't use my arms to type/mouse all day. I spent some time researching speech input for coding a year ago, and found a few fledgling solutions, but I'd need to expend a lot of time to get something remotely workable.
Your idea is, if I may say so, genius, and having that as a backup sets my mind greatly at ease, so thanks.
On pairing in general: For a long period of time when I worked at bigger companies (before co-founding my current venture) I pair programmed 90% about of the time.
For the most part I really enjoyed it, we alternated pairs quite frequently and I worked with people at a range of seniorities. I really enjoyed alternating between playing the role of teacher vs learning from someone much better at the (insert programming language, database, business domain).
I think during that time I also had some of the most stimulating and deep conversations about software development. When pairing, pairs have a deep shared context that's hard to replicate in other scenarios.
There are some drawbacks though, the main one being it's extremely full on and emotionally tiring. I'd class myself as a high functioning introvert and after a day of pairing I needed some serious regenerative quiet time!
My advice to someone considering this suggestion would be to try some pairing now with friends or colleagues and get a feel for it and see if it works for your personality and working style.
Hah, I'm stealing that phrase to describe myself too!
I've never "formally" done pair programming, but have sat and coded/reviewed with colleagues for short stints, and generally found it OK.
From time to time I have to do full days (or several days) of workshops, and find those really tiring and emotionally draining. I always think I fare better 1-1 with people though, so if pairing was a necessity, I'm sure I could adapt.
I touch on this in the article, but I should say: I'm an edge-case when it comes to CTS. Most cases resolve spontaneously. Part of it is that my nerve dislocates when I bend my elbow, and this mobility causes a bunch of additional friction / inflammation. This happens naturally to about 13% of the population, and mine is particularly pronounced.
All this to say, if you ever do start to get a burning or tingling in your elbows or wrists, or get numbness in the hands, it'll likely go away on its own, or with conservative treatment (a physical therapist is a great first step!).
And, for those who also fall into my unlucky percentile, hopefully it helps to know that there are tools and a community that exist to let you keep working without needing to use your hands at all!
The scare got me very interested in accessibility and I've looked into using Talon with eye tracking and stenography via Plover to improve my ergonomics, but I've never fully committed to any of them, but I'd love to know what communities exist. A Kinesis Advantage and a vertical mouse seem to keep it at bay on all but the longest of work days.
This doesn’t surprise me; Talon is a lot like Vim.
They’re both modal: Talon’s command mode roughly matches Vim’s normal mode, and its dictation mode roughly matches Vim’s insert mode.
Talon’s ordinals apply to Vim operations (normal mode) as well: operations can be preceded by a number which normally says how many times to do it. Talon’s “go left ninth” would be 9h or 9← in Vim. (h goes left one character.) “One zero third” is more cumbersome because in Vim it entails switching between modes (Talon and Vim modes don’t match precisely), but starting in normal mode, a1<Esc>3a0<Esc> works to append 1000 where the cursor is. (a to enter insert mode after the current position, 1 to type a 1, Escape to return to normal mode, 3 to say “do the next thing three times”, the next thing being appending a zero.)
Both Talon and Vim value this sort of consistency in being able to build bigger things from smaller pieces.
So I imagine Talon would work quite nicely with Vim, once you get talon to shift out of the way in some cases and let Vim handle those things. (It’s similar with browser extensions that let you interact with the browser Vim-style: when the page in consideration has extensive key bindings, especially Vim-like things like j/k for down/up, the extension actually gets in the way.) I expect you’d find that having used one made learning the other a good deal easier, since they’re working in just the same ways.
My interest currently is in voice-assistants and developing something like a personal assistant / code-monkey / grad-student.
I did have a stroke a few years ago, from which I've recovered, but it made me realize that almost all of my work, hobbies, aspirations, are tied to fairly extensive hand-eye integration. It's good to start building a Plan B before I need it.
On a(nother?) side-note, my 88 year old dad still codes and his experiences have really shown me how bad we are at accessibility, as a developer community.
One final thing -- watching you work in this way highlights for me how low-level a lot of the stuff we do as developer is. There's a certain amount of just text massaging that seems irrelevant to what we're actually trying to get done (although it can be fun/soothing/aesthetic).
EDIT> The last point is why I can't give a fark about Vim/Emacs/whatever debates and obsessing. It seems like it might be fertile ground, but IMNSHO it's a trap.
There are moments where I think this is going to be the future of programming, since code as text is only the easiest for as long as typing is the easiest way to record things unambiguously.
But for the most part it’s still pretty frustrating. If ASR systems can get the sentence error rate down by an order of magnitude or two, I sincerely think this will take off not just for accessibility, but for normal use.
Until then, it’s a PITA that is saving my career nonetheless.
[0] https://caster.readthedocs.io/en/latest/
My impression of talon's speech recognition is that it's good enough for voice coding, but could be improved when it comes to dictating English prose. That said, there are promising avenues of improvement. If you pay for the Talon beta, there's a more advanced voice recognition engine available that's much better at English, and you can also hook into Chrome's voice recognition for dictation.
More specifically, "can work" means: if you run Talon on Windows or Mac alongside Dragon, and don't configure Talon to use its own engine, Talon will automatically use Dragon as the recognition engine.
Unfortunately it looks like Dragon on Mac isn't a thing anymore, which is a shame because my main dev machine is a Mac. Or at least it used to be, before I needed to start dictating everything.
And I do have some users voluntarily switching from Mac Dragon to the improved speech engine in the Talon beta. Mac Dragon has been kind of buggy for the last few years so you're not missing much.
On the Dragon side, you need to uncomment and update the `[[socket]]` section to listen on an accessible IP.
On the client side (Mac in your case), uncomment / update the `[[remote]]` section to point at your other machine.
You also need to make sure both configs have the same secret value.
From there, restart Dragon and look in the Talon log on the Mac side for a line like "activating speech engine: Dragon".
To prevent command conflicts, I recommend setting Dragon to "command mode" (if you have DPI), and only adding scripts to Talon on the Mac side.
If it doesn't work, you can uncomment the `logfile` line in draconity.toml on the Dragon side, restart Dragon, and look in the log to see what's going on.
EDIT: Seems like caster itself has instructions for an open source recognition engine to pair with it. Not sure how accurate it'll be but I'm going to give it a shot!
I hate the idea of relying on closed source software to be able to continue in my profession. If this works I'm definitely going to be donating to the FOSS options
Talon ships with a libw2l.so/dylib on Linux/Mac built from my open source repos.
I didn't know that there was a pytorch implementation of the w2l architectures..
For my own use I’ve already built Pytorch and CoreML frontends, with a shared model format (can convert models to/from wav2letter format and my custom format), and I have the ability to create new models in these frameworks from wav2letter architecture files.
I still run my training in the wav2letter framework, but for compatible training in Pytorch I would mostly just need criterion implementations. I assume warpCTC is fine for the CTC models. There’s also a third party Pytorch ASG criterion package but I haven’t tried it yet.
I'd like to try to record my usage of Dragon that I could fine-tune my own model, but it's harder to get around to hobby coding projects like that now that coding is more of a pain in the ass.
The unfortunate reality in my case is that using Dragon is the least frustrating way to keep working. I don't think closed paid ASR models will stop being noticeably better better than three open ones until the state-of-the-art error rate is basically zero.
I do like that caster is open source where talon isn't, but preferences like that are pretty low priority when say your career is on the line.
For sure if its on the line right now you use what is available and works best right now, but the reality of niche software is the companies/people backing it tend to close down or move on as they realize the market is too small to sustain them. Or Microsoft updates its OS in a way that forces them to expend considerable time making it functional again and they just don't have the bandwidth for it so you're stuck for some amount of time or indefinitely.
If you have no other choice, its definitely better than nothing and you want to use what enables you to do your job the best. But in terms of where I'd be willing to throw my financial support? It'd need to be something for the better of the coding community as whole. For me, this means an open source tool or set of tools, not a proprietary system.