Ask HN: Is there any work being done in speech-to-code with deep learning?
Is there any work being done on speech to code in a deep learning area . I have severe RSI which prevents me from coding at all . I have tried to use speech recognition software such as vocola and windows speech engine . but it required me to speak in such a way that I always would hurt my throat . I have also injured my throat multiple times so I am searching for a solution that is more conversational then command driven . I have written over 10000 lines of command Fargo Cola and they're still too many edge cases which require me to continually speak in an Abrupt manner that causes strain on my throat .
62 comments
[ 4.6 ms ] story [ 94.4 ms ] threadFood for thought for sure. Good luck.
To me it seems like learning how to talk to Alexa or Cortana or "Google" is a limitation or regression for humans. This shows that it could actually be beneficial.
Thanks for this philosophical rabbit hole just in time for a weekend.
I have found this research paper[0] that describes using a machine translation with a CNN and a r n n. it seems one of the main problems that come up with is a lack of training data so to progress this further it seems like we would need a big corpus of parallel text between natural English describing what the code does in the actual source code. I only have some novice experience with training machine learning models through transfer learning with fast AI but I have found a repository of tutorials[1] that teach you how to do machine translation between two languages and it seems maybe this approach might be applicable. There is a small corpus of text[2] with python code with annotations but and the paper itself at States. It probably needs more to get a higher than a 74% accuracy.
the only thing I can really think is to scrape GitHub make a website where you can get people to crowdsource annotations for code samples. I'm not sure on how to deal with how much and how little code would be for a sampling. I eat how much of the code would you need for a good training sample versus just using isolated functions. Perhaps that something like this might be used in correspondence with GPT.
[0]https://www.groundai.com/project/machine-translation-from-na... [1]https://github.com/bentrevett/pytorch-seq2seq [2]https://ahclab.naist.jp/pseudogen/
EDIT: Actual demo with coding starts at 18.00: https://youtu.be/YKuRkGkf5HU?t=1076
[1] https://www.youtube.com/watch?v=YKuRkGkf5HU
The best project I've seen for voice coding is Talon Voice, but I doubt anything novel is being done with it and deep learning. I'd suggest trying it out if you haven't. They also have a pretty active slack channel, you might have some luck asking them if they know about anything on the horizon.
https://talonvoice.com/
I'm definitely open to incorporating deep learning directly. I've already signed up for the GPT3 API waiting list and I have some ideas on how to use it, and I generally have some ideas on how I might otherwise approach more natural feeling voice programming down the line.
That in combination with switching to a Lisp (Clojure) almost made it feasible for me to code with RSI.
I just became a manager instead because I couldn’t work from home and talking like that in the office was a no-go for me.
If that’s your cup of tea you’d be surprised at how happy upper management is to have someone who’s actually good at technology be willing to engage with them.
Check out this success forum of people who have healed from all kinds of chronic pain symptoms by dealing with stress and changing their mindset:
https://www.tmswiki.org/forum/forums/success-stories-subforu...
Rarely does the RSI flare up when I'm doing something I enjoy. Stress is definitely a component.
I haven't seen any demos actually using speech recognition as the input, but I'm excited to try that as soon as I'm off the waitlist. One of the caveats I have in mind is that speech recognition language models are going to be worse than GPT3, so the best option might be getting GPT3 to do the language decoding in-domain _and_ do the task. (The other caveat I have is it seems pretty slow for interactive speech use, but I have some ideas on how the knowledge could be cached in a sense over time.)
I develop kaldi-active-grammar [0]. The Kaldi engine is state of the art for command and control. Although I don't have the data and resources for training a model like Microsoft/Nuance/Google, being an open rather than closed system allows me to train models that are far more personalized than the large commercial/generic ones you are used to. For example, see the video of me using it [1], where I can speak in a relaxed manner without having to over enunciate and strain my voice.
Gathering the data for such training does take some time, but the results can be huge [2]. Performing the actual training is currently complicated; I am working on making it portable and more turnkey, but it's not ready yet. However, I am running test training for some people. Contact me if you want me to use you as a guinea pig.
[0] https://github.com/daanzu/kaldi-active-grammar
[1] https://youtu.be/Qk1mGbIJx3s
[2] https://github.com/daanzu/kaldi-active-grammar/blob/master/d...
I'm eagerly awaiting this! If I wanted to try to get something working now, I'd need to invest a lot of time - being able to get started quickly would be amazing.
The Kaldi engine, being developed primarily for research in speech recognition, can support a huge variety of "models". I think the consensus general best for most use cases (particularly for real time, low latency, streaming use) currently would be considered to be the "nnet3 chain" models, which are what my kaldi-active-grammar uses/supports.
Basically, my question (and I assume many other users') is "I run <Linux/Windows/Mac OS>, what are my options and how good will my recognition be with each?". Your answer above helps, but it doesn't entirely satisfy me, as I'm not sure if a model is the recognition engine, or if the engine uses the model, or how I can use it, etc.
Also, talk to an ergonomics person about it, and it sounds like notebooks are out at this point unless you have an external keyboard, mouse and monitor.
In my experience, I've found any services claiming to do deep learning produced far worse results than what we could get with simple approaches. That is, when faced with non-grammatical sentences (or rather, sentences with a different grammar than English's). Of course that's because models are not typically trained with this use-case in mind! But the fact that you need a huge load of data to even slightly alter the expected inputs of the system, to me, was a deal breaker.
For the specific case of programming with voice, Silvius comes to mind. It's built and used by a developer with this same problem. It's a bit wonky having to spell words sometimes with alpha-beta-gamma speech, and it won't work without some customization, but on the other hand it's completely free and open source: https://github.com/dwks/us
That said, Silvius is more of demo than a product, the IMO best voice programming options right now are (in alphabetical order):
- Caster/dragonfly (fully open-source if you use daanzu's Kaldi engine, which is way better than Silvius afaik, I think even the creator of silvius uses dragonfly with dragon instead of using silvius)
- Serenade (fully commercial, I haven't looked at it much recently but biggest caveats afaik are accuracy, the fact speech recognition is web based, and it's restricted to specific languages and IDEs while caster/talon are for full system control and not just programming)
- Talon (my project, semi-commercial as I work on it full time and draw income from it but aim to give all necessary features away for free, some benefits include a fully offline and open-source speech recognition engine, and I have other bonuses like eye tracking and noise recognition)
Not yet, but will do, thanks!
However, I'd still be hesitant to build a product on top of that: Does voice to bash help us if we now want to do, say, voice to python? At least we'd need to re-train the system with completely new data, and even if we use transfer learning to our advantage, it's not an easy task. There's also no guarantees that the chosen neural network architecture that works for bash, will work the same for any programming language (think of a radically different syntax, like Lisp for example).
The training must also be re-done for any variation in the input format to some extent. i.e., accent, expected background noise levels, and of course (human speaker) language.
ML has its use case, but I typically see these nice demos as that, demos. When you have to build a real product and solve user problems, you can't rely on a black box doing what you want.
Also they do have a related Python demo: https://news.ycombinator.com/item?id=23507145
Speech is a completely different stack to this, but honestly (english) speech is much more of a solved problem here than general knowledge.
There are a few good OSS offline deep speech libraries including Mozilla DeepSpeech [3], but their resource footprint is too high. We settled on the currently less mature vosk [4], which is based on Kaldi [5] (a more popular deep speech pipeline), and includes a number of low-footprint, pretrained language models for real-time streaming inference. Research has shown how to deploy efficient deep speech models on CPUs [6], so we're hoping those gains will translate to faster performance on commodity laptops soon. You can follow this issue [7] for updates on our progress. Contributions are welcome!
[1]: https://github.com/OpenASR/idear/
[2]: https://cmusphinx.github.io/
[3]: https://github.com/mozilla/DeepSpeech
[4]: https://github.com/alphacep/vosk-api
[5]: https://github.com/kaldi-asr/kaldi
[6]: https://ai.facebook.com/blog/a-highly-efficient-real-time-te...
[7]: https://github.com/OpenASR/idear/issues/52
And one shameless plug deserves another! Vosk is a great project, but my kaldi-active-grammar [0] (mentioned in another comment here) also uses the same Kaldi engine, but extends it and is designed specifically for this use case. It supports defining many grammars, in any combination, and activating/deactivating them at will instantly per-utterance. I think it's probably a better fit as a backend for your project than vosk. My work focuses on the backend technology, so it would be great to have more front ends using it to put it within users' reach (so to speak).
[0] https://github.com/daanzu/kaldi-active-grammar
Also, I bought a keyboard tray that supported a deep negative angle, which helped me keep a very anatomical (relaxed and natural) position.
Also, figure out that mouse, somehow. Something like the above, plus switch sides frequently.
I've no idea if that could help you, but after a few years, I'm largely in remission.
I know this isn't really what you were asking, but I'm somewhat hopeful you can find relief. Good luck.
Can you use a touch screen or mouse? I went ~13 years without using a keyboard, and typed with mice (some customised), trackballs, and touch screens, mostly using predictive typing software I wrote. In that time I did a lot of programming, including a whole applied maths PhD.
One of the best mouse setups I came up with a variety of versions of was moving the cursor with one hand, and clicking with the other. Holding the mouse still to click the button accurately is a surprisingly problematic movement. I made a button-less mouse with just a flat top to rest the side of my hand on, with a bit sticking up to grip. Standalone USB numeric keypads can be remapped to mouse clicks and common keys.
Touch screens can also be very good, if set up right, as all the movement can come from the big muscles and joints of your upper arm and shoulder, and your fingers and wrist don't need to do much. The screen needs to be positioned well, not out in front of you, but down close and angled in a comfortable position to hold your arm for long periods.
When using commands at my computer, I frequently find myself muttering and grunting things that even I think to myself (that is utterly un-understandable), yet the computer understands just fine. Dictating for prolonged periods can be tiring for me, but I can happily code by voice commands all night.
More info in my other comments here: https://news.ycombinator.com/item?id=23507363 https://news.ycombinator.com/item?id=23507829
I always wanted to learn vimspeak: https://news.ycombinator.com/item?id=5660633
I am not a coder, I am a writer. I wonder why all these AI people are trying to create things that will displace my means of earning a living instead of something that will create applications?
Why can't I tell my Mac: "Computer: take this collection of files and extract all the addresses of people in Indiana."
AI writing is currently terrible. If your writing income can be replaced by AI, you're likely summarizing simple topics (e.g. unemotional, analysis-free facts of a recent event, like a change in a stock price). In that case, your job isn't safe anyway, because there are $2/hr humans in other countries that can also take your job.
Note that all software is capable of eliminating jobs, whether it's automating writing or something else.
> Why can't I tell my Mac: "Computer: take this collection of files and extract all the addresses of people in Indiana."
NLP is nowhere near being able to generally understand requests like this. Try talking to your Google Assistant and doing anything other than a simple lookup. It fails spectacularly.
Think about all the cognition in this task: first it has to understand what you're asking it. Then it has to figure out which collection of files you mean. Then it has to understand what an "address" is and what it looks like. Finally, it has to have a concept of "Indiana" and an understanding that an address can be in Indiana.
In terms of effort, it's far more efficient for you to learn the simple programming skills to do this yourself (or to use a purpose-built tool) than to create AI that does this generally, which will cost at least billions of dollars more in research time.
The tough part is really probably the implicit knowledge of what the database columns are called. Is Indiana a "state" field or a substring in an address field? Depends on the database.
This mixing of "soft" ml queries with "hard" requirements is a really interesting space, which I think will produce massive results if we can get it under control. It's similar to the problems in the AI dungeon master space... And probably tutoring, as well...
You can have substantial success in transforming English to French without understanding what is said. Some big grammar rulebooks, some dictionaries, and a few terabytes of sentence-aligned parallel texts and data on statistical colocations in the languages will get you most of the way. It's far from 100% accurate but the human brain is surprisingly good at filling in the gaps with the mangled grammar and odd turns of phrase.
I'm not so sure the same is true about English to SQL query. The exact meaning of the English must be understood, in context, in order to construct the query correctly. No room for error or fuzziness. The database engine doesn't have human cognition to fall back on should the query be slightly malformed or ambiguous.
I think this has to do with overestimating the power of current ML techniques combined with severely underestimating the complexity of jobs like writing. This tends to result in solutions that "overreach" and end up disappointing. If we scale back a bit, we could build some really fantastic tools that work with people.
One great example of this is the movie Spiderman: Into the Spiderverse. In order to draw the comic-book lines on faces, they used a tool that would learn over time from the artists themselves. It would guess, the artist would adjust, it would learn. This is the perfect fit for ML: try to help the human, not replace them.
Ask HN: I'm a software engineer going blind, how should I prepare? (https://news.ycombinator.com/item?id=22918980)
So first, I switched my mouse to my non dominant hand ( left hand for me ), as that hand already has many things to deal with. I'm also using a workstation that allows me to mount my displays at eye-level while sitting or standing. Not hunching over is ergonomics 101. Second, I switched from a standard keyboard to a split keyboard. I tried many -- Goldtouch, Kinesis Advantage2, Kinesis Freestyle -- and ultimately settled on the Ultimate Hacking Keyboard.
I could write many more paragraphs on how I customized it and why it won out, but the most important thing is that is is split and it "felt" best, once I mastered the key placements ( arrows are in different places ).
Third, I started learning VIM. Vim is really awesome but up until recently didn't have great IDE or other editor support. Now it does so there's no reason to not use it. I mostly use it for quickly jumping around files and going to line numbers.
Fourth, I'm always looking to optimize non-vim shortcuts in my editor. For example, expand-region ( now standard in VSCode ) is one of my favorite plugins.
Fifth, I'm very conscious of using my laptop for long stretches of time. Mousing on the mousepad is much more RSI inducing than using a nice gaming mouse and the UHK keyboard.
All of this to say that RSI doesn't have to be career ending. If you're doing software work and you have functioning hands and wrists you should definitely look to optimize typing before looking to speech to code. Good luck!