Very interesting - I was sort of expecting it to happen soon.
I have been playing with using Whisper + Github Copilot in Vim [0]. The Whisper text transcription runs offline with a custom C/C++ inference and I use Copilot through the copilot.nvim plugin for Neovim. The results were very satisfying.
Edit: And just in case there is interest in this, the code is available [1]. It would be very awesome if someone helps to wrap this functionality in a proper Vim plugin.
If this works well, I would pay a seriously high amount of money. My daily coding time is currently limited by the pain in my hand/fingers that eventually becomes too uncomfortable, and I have to wait for a "cooldown" period of days to "reset" my hands back to normal. I can't even code on a normal keyboard or trackpad for a long time anymore.
The problem with current voice programming systems is they're just too slow so I end up getting impatient and using my fingers anyway
I imagine you have done extensive research on your own, but in any case I found this article by Josh Comeau on coding with voice commands and eye tracking very interesting: https://www.joshwcomeau.com/blog/hands-free-coding/
An ergonomic keyboard layout like Workman doesn't hurt either. QWERTY was made with typewriter jamming in mind, sacrificing ergonomics completely, especially for English.
Have you tried an organ MIDI pedalboard and a script to translate MIDI to keystrokes? You could also put a micro controller between the pedalboard and your computer so that it looks like a normal keyboard to the computer. I do not know whether that would be pratical, but some sort of feet keyboard is in my idea space for what if.
Drop the "Hey Github" nonsense (hopefully it's only for illustration purposes anyways) and … this will be a generational paradigm change in how to write code… if it works. The hard part will be editing code with your voice too. Like "no, I meant …" etc.
VERY PROMISING, in any case you can just manually fill the gaps with the keyboard!
I could see it maybe being important once github codepilot is embedded in it? You tell it roughly what you want and then adapt by hand. But it is kinda funny seeing parent make such claims so early
I don't want to disparage their work, because it's really impressive, but "fill null values of column Fare with average column values" is closer to AppleScript than it is to natural language.
It solves the issue of trying to speak obscure code syntax like “close parenthesis semicolon newline”.
That’s enough to lower the barrier to entry for many people; I don’t know how good it is practically but it’s disingenuous to suggest it’s not offering a novel solution to an old problem.
> The problem with speech to code has always been that precise syntax is hard
The biggest problem is that talking sucks. You presumably can handle voice input as well as is possible, yet here we are typing to you anyway, and for good reason. Even if the natural language part is nailed, you may as well type in that natural language.
I imagine it will bring some quality of life improvements to those with certain disabilities, but I don't see why the typical developer would want to go in that direction.
The example puts it quite well. You kind of know what you want to achieve, step by step, but are not so comfortable with your tools.
Usually this kind of exploratory work involves a lot of Googling and copy-pasting snippets from Stackoverflow without putting too much time in trying to deeply understand things. If you get out what you want - great, if not, back to Google.
It seems wonderful for people who can't as easily use a keyboard, but for most people, this doesn't seem any easier than using a keyboard. Am I missing something?
If voice dictation was a killer feature, everybody would use it all the time for ordinary texts. But for some reason only few (lawyers? doctors?) use it.
I believe that's mostly because it doesn't work reliably. Doctors, lawyers, architects etc have a somewhat limited professional vocabulary and often say the same things, so voice recognition works pretty well for them. But when you write a random message, you have a much broader range of topics, and dictation that fails quickly makes the whole thing change from an improvement to an ordeal. "No, not 'or deal'. delete word. delete word. delete word. O-R-D-E-A-L. Yes, that's it. No, don't write that, sigh".
I use a Czech keyboard layout on my Mac, because Czech has some letters that don't exist on a US keyboard, and I don't like switching between layouts. So basically all "programming" characters (braces, brackets, parentheses, apostrophes, quotation marks, pipes, colons) are behind modifiers.
I would totally enjoy being able to tell my IDE to "call foo with bar and string hello there end string with a block of gee times two" or something, instead of:
foo(:bar, "Hello there") { |gee| gee * 2 }
Just that, not having to think about typing different symbols would be a serious quality of life feature for me.
> I don't see why something similar can't be done for the Czech alphabet.
It probably could — we already can't fit all the letters with diacritics on the number row, so "ď, ť, ň, ó" are key combos. But as far as I know, Czech uses diacritics a bit more than Polish (e.g. for sounds that are digraphs in Polish), consider:
"Že se nestydÍŠ, nutit lidi psÁt ČeskÉ speciÁlnÍ znaky pomocÍ dvojhmatŮ!" — that's 10 modifiers just for the diacritics.
Having ALL diacritics as modifier combos would make typing actual texts even more annoying than programming is now.
Have you tried the UCW layout? English-like keyboard, but with a bonus modifier key that produces Czech (and other) letters. I use it and it's so much better than the traditional Czech layout.
If it works equally good like Apple Siri or Google Hey (or whatever its's called), then it will be ... totally useless? I can't imagine that they bring a better product than two of the richest companies in the world even can't figure out (perfectly). And if I need to read and adjust all my code for typos, I can just write it myself.
Because in my experience it is very often like "Call Peter" -> "Today it's sunny in NY".
> I can't imagine that they bring a better product than two of the richest companies in the world even can't figure out
Are either of those companies investing particularly heavily into voice agents? Certainly neither of them has anywhere near the kind of power of something like Copilot.
Also, a general agent is way different from one that's specific to writing code.
To be fair Siri was really good before iOS15 on the phone - very rarely got a word wrong then I don't know what they changed but it went belly up for me and many other people have said the same.
On macOS it still seems pretty good - I have carpal tunnel syndrome and by Thursday or Friday most weeks I end up using Siri to dictate not code but a lot of conversations in Slack, pull requests, iMessage, etc. In fact, I wrote this reply with Siri right now.
I definitely notice there's a difference in quality depending on your network latency I thought quite a bit of the processing was done locally now, but latency seems play such a part in its quality.
iPhone ability to convert speech to text has always been good. It’s always been Siri’s capacity to take a meaningful action from the recognized speech that has been problematic.
>I can't imagine that they bring a better product than two of the richest companies in the world
Code is much more constrained by language syntax though.
Even for the "call peter" example, while the input is easy, the expected range of inputs that Siri should handle and be able to differentiate it from is huge.
Of course this is still a problem for e.g. defining variable names, where you could say anything.
I've been trying to use Siri while driving more and more, it's amazing how distracting it is compared to peaking at the screen (it's naughty, I know, I try not to do it).
But yeah, something about talking to a device which gets things wrong all the time is ridiculously distracting, at least for me.
Sometimes I look back at the road after trying to workout what it interpreted and I feel scared how focused on the phone I became.
Generational? Idk. I work for a company that regularly sends out surveys, and there are several tools to integrate voice into it. Willingness to speak instead of type is quite low across respondents (which is a representative population sample). It looks as if speaking to a machine does not hold the same appeal as speaking to a human (something that can also be seen in telephone queue screener questions).
This is not marketed as an alternative input mechanism for people who have otherwise no difficulty typing code. It's an input mechanism for people whose abilities to type are limited.
I hate talking to machines. Sometimes it’s the best option (I love using a voice assistant in the kitchen), but almost always I’d have a full keyboard as an interface instead.
If machines were amazing at Speech-to-Text, okay, sure. But while the capabilities are impressive, they still kinda suck at it.
The only voice control anything I use is to create reminders on my iPhone, and the only reason I use that is because the default reminders app's UX is really bad that it's quicker to use the voice commands.
I don't see how text to code would be faster than typing. And even if it is, typing speed is not really a limiting factor in the speed at which I can produce code.
The first amazing result would not be for you to program with this, but for somebody with a phone to be able to automate a few small tasks with just the voice.
It got "Theresa's and Aidsdorm" for "Turisas and Alestorm". Surprisingly, it got pretty close with the German band Schandmaul (something Alexa recognizes 100% of the time as Sean Paul), transcribing to "Schandmauel or Schandmöhrl".
But yeah, that is pretty close to amazing.
I kinda forgot about it after seeing that the Rhasspy community experimented with it, and it had issues with short utterances and a slow startup time.
Exactly. We've seen a constant improvement in the tech for decades. I remember before color lcd phones that they had "voice control" and today we have assistants, which are orders of magnitude more sophisticated.
Yet, it hasn't stuck. I'm exclusively using Siri to set timers. Most people are like me, or don't use it at all. Some use assistants for googling factoids or something. Fidelity wise, it's really underwhelming.
It's not a social acceptance issue, because people would still use it at home, and they don't. It's a small chance there's some key UI insight missing (discoverability for one), but I doubt it. Even with perfect UI, natural language is quite flawed when you're dealing with technical details (see exhibit on variable naming).
Anyway, the chances of Github solving this in an exceptionally difficult subdomain, as a side project, seems like a... Let's say, long shot.
That said, the silver lining in all these billions spent on voice interfaces is accessibility. For some people, these things are a life saver.
I dunno, we already have stuff like Krisp AI background voice cancellation. I don't think it's far away to completely cancel background talking out. This is already huge for things like pair programming while one person's in the office, one is at home. If you have noise cancelling headphones for the person in the office too (with a bit of white noise), you can have a pretty perfect call in a noisy room. (not sponsored)
I'm not bothered by my call quality, which as you noticed is fine, I'm bothered by all the other people speaking (sometimes quite loudly) on their calls while I'm not on a call :-)
It's the future I always imagined as a child. A vast divider-less cubicle scape of people in Patagonia vests who define all caps constants by yelling at their standing desks.
"USER!! UNDERSCORE LIMIT!! EQUALS TWO THOUSAND AND FORTY EIGHT!"
"… this will be a generational paradigm change in how to write code… if it works."
Why?
Can't really see myself working like this in an office, plane, cafe, with music on (my favorite way to code), in the house where my partner is also working. Then as others have said, editing might suck.
To note, there's a class action lawsuit against GitHub Co-Pilot since it learns from a bunch of open source code with very specific licenses. It's very interesting from establishing copyright in an AI training perspective. Hopefully it goes the distance and some nuanced arguments come out in the court case.
People can’t seriously believe this is going to be useful at all?
I can see this helping as an accessibility tool, but beyond that I don’t think it will be useful. This kind of assumes you know everything about what you’re doing, most of the time you don’t.
I remember a talk given some years ago by a man who was using voice to text for creating source code. The key point I remember from his talk & demonstration is that it was not casual ordinary speech but instead a very weird mashup of sounds intended to represent the various symbols which we use in source code.
Spoken language is incredibly ambiguous. It's one thing to generate a drawing which can vary wildly in output and still be acceptable. It's another to specify something precisely to a computer. Working with non-programmers on a daily basis it is incredible how difficult it is to communicate even relatively simple things without confusion.
So all the more power to them, but I am very skeptical. Especially since co-pilot has zero knowledge of the formal semantics of programming languages.
This is a lot different than the half ass auto complete that it already does since that at least has some context.
With copilot ambiguous language gets transformed into concrete syntax. If the implementation doesn't fit your ambiguous request, you should be able to refine … with ambiguous language. Theoretically this would create a "programming dialog" environment.
So you are going to have to verbalize something, interpret the code, build a mental model of how it works, and then if it does not match what you want go back to step 1?
That sounds exhausting when we have spent countless human years developing languages which let us communicate our intentions precisely to computers.
If you don't do this there is no ambiguity detector. Meaning it's entirely possible for the computer to interpret what you are saying completely different than intended, yet it is a perfectly valid interpretation. So the only one who can qualify if it got it right is you.
It's the same with copilot; you have to know how to implement things to implement things with copilot (for the most part), but when you are a programmer and you could write the code, then you know the prompt to write to generate 10+ lines of code for 1 comment of text all day long. Especially for data transformation, copilot has been a real magic tool; if you put in a comment:
/*
this functions transforms this json from:
{
... some complex structure in json
}
to this json:
{
... some different structure in json
}
*/
... copilot comes up with the function that takes in the first and spits out the latter. Even if the fieldnames do not match etc, it usually 'guesses' right what fits on what (so it does have some context from it's learning phase what 'looks alike' or 'might be the same thing'. Example: I had a structure with firstName: string, lastName: string and a target structure with name: string; it just did name: firstName+' '+lastName, which was indeed what I wanted. But it comes up with more intricate stuff as well that is pretty much surprising (too human basically).
What is another bonus; if you generated function transfromAtoB(a: A) above, then you only have to do:
/*
do the reverse of function transfromAtoB, accept json structure B as input and return structure A
*/
And it'll come up with the reverse.
It's not hard to write yourself, but it's boring and error prone (some of these structures are huge). Now I press tab a bunch of times, and run the tests to see if it worked. I am also not that worried i'm infringing someone's open source code; this is all way to custom to look like anything else. That's where this shines; things where it verbatim copies something, you should've been using a library anyway.
Statically typing and using typescript definitely works better than other combinations I have tried (C# was pretty bad last I tried it, JS is good but often subtly wrong because of type issues).
I spent half an hour today trying to convince the O2 voice agent to get me a real person. Conversational AI is a special kind of hell filled with unhappy paths.
But for a glimpse of the future watch The Expanse or read William Gibson's Agency.
It does look like we've made some progress in the 15 years since. I do wonder how this would work in an office setting though - so much noise, so much distraction, and so much crosstalk between programmers...
Hahahah thank you for posting this, I was about to go look for this because I remember being in tears laughing when I saw it this first time and immediately thinonof this whenever I see voice controlled things
From memory, there was a time (end of millenium?) when using voice recognition to write documents was the next big thing. There was a pricey bit of software for Windows that was popular with power users and they would spend hours training it to their voice.
Then it seemed to just die off. I don't think it was bad technology, because I don't think novelty value was enough to account for its popularity - you had to put hours in to get it to work well, it wasn't a casual toy.
What's changed since then in terms of technology? Unless it's very significant, I suspect it will go the same way. Apart from an assistive technology viewpoint, my gut instinct is that it's not that satisfying or rewarding talking to a computer all day.
Not a doctor, but I also started to using the dictation feature on my iPhone more and more recently. It's often more convenient than typing when I'm walking (which I do a lot) and the pickup of voice messaging, talking to your phone like it's a mic, made me more comfortable doing it in public.
one thing that changed that you can see in a demo, that it's speech recognition paired with a neural network. He doesn't have to say "write titanic = titanic.dropDuplicated()" he just says, "drop duplicates from titanic" so you "in theory" have to write less. It probably falls apart when you have more complex things to write and then you have to fallback to speaking it out word for word, but it is an interesting development
Programming languages need to be unambiguous, but voice interfaces to them don't have to be. With a voice interface, you can say something vague, see what it gets translated into, and either click Run or edit it.
WFH become more common. You cant program with voice in office full of people who also program with voice. At home you can multytask work with voice while doing home stuff with hands. You just need projector do display on wall while you cook or assemble IKEA table.
I sometimes use voice recognition in Notes on my Mac to write up my meeting notes, but I find that my waffly speech results in very verbose notes.
Also still quite a few mis-hears where I read my notes back later and have to work out what I was actually saying.
Can I have transcription that can then turn my rambling into neat and concise prose?
Ah yes, Dragon NaturallySpeaking. Training for hours and hours and getting incredibly subpar results. It was a fun toy but there's a reason it didn't really take off in corporate settings.
Dragon NaturallySpeaking is still alive at least in medical practice. Its Nuance Dragon Medical One product is fairly popular in some regions for medical report dictation as radiologists don't like to write them down (sorry ;) ). I've seen a Philips product in that field too. Seems like LG's TVs used their recognition engine for a while (don't know if that's still valid)
The story for the most mainstream-popular dictation softwares is kind of funny. Back in the late 90's there was Dragon NaturallySpeaking and IBM's ViaVoice. In early 00s, after a financial fraud and bankruptcy involving both the then current Dragon owners and Goldman Sachs, they got bought by Scansoft. Scansoft bought Nuance, began to use its name, and then got exclusive rights for ViaVoice (!) from IBM.
Now, in March this year, Nuance has been acquire by Microsoft.
> What's changed since then in terms of technology? Unless it's very significant, I suspect it will go the same way. Apart from an assistive technology viewpoint, my gut instinct is that it's not that satisfying or rewarding talking to a computer all day.
Training data is now abundant compared to twenty years ago, and so is computation power. That means training can be much more complex now.
The underlying technology is now typically neural networks (broadly speaking), whereas twenty years ago it might have been Hidden Markov Models.
Overall, recognition quality, even without speaker-specific training, is now on a very different level than back then. Whether it’s considered good is a matter of opinion. But it’s significantly better than twenty years ago.
Isn't that a very niche application simply because it's voice-based? I.e. it can only be used if you are alone in an office, otherwise you would be annoying your coworkers and the voices would get mixed up.
The only time I’ve ever successfully used voice recognition was teaching skyrim to recognize my words of power. Shouting fus-roh-dah ! was incredibly satisfying.
> What's changed since then in terms of technology? Unless it's very significant, I suspect it will go the same way. Apart from an assistive technology viewpoint, my gut instinct is that it's not that satisfying or rewarding talking to a computer all day.
It's hugely significant - look at this graph of Google's speech model accuracy across 2013 to 2017:
1) Improvements in speech to text, as others have mentioned
2) Improvements in language models (and model size) allowing for more flexible interpretation of speech. This isn't dictation anymore. It's more like instruction. You don't have to tell the computer exactly what to write, you tell it in much more broader terms. Eg "pull this out into a function". Or "delete the cookie before creating the transaction". Or "lint the file".
That's my guess anyways! This mostly feels like a voice interface to Copilot in a lot of ways. Can't say whether it'll be effective, but I'd love to be able to program while I'm e.g. on a stationary bike!
When I talk to my Google Home then 50% of my brain power is engaged in predicting and working out how to best phrase something so that the "AI" understands what I mean and the other 50% is used to actually think about what I want to accomplish in the first place. This is just about okay for things like switching lights on/off or requesting a nice song I want to listen to, but I could never be productive programming like this. When I'm in the zone I don't want to have to waste any mental capacity on supplementing an imperfect AI, I want to be thinking 100% about what I want to code and just let my fingers do the work.
For that reason I think this will be less appealing to developers than GitHub may think, otherwise I think it's a cool idea.
It's really useful for those who have challenges typing (arthritis, disabilities etc..), perhaps not best for general audience as typing with auto complete is faster.
The assistants (Google, Alexa, Siri) are not great at NLP. Compare how you speak to them vs speaking to a LLM like gpt-3, there is a world of difference. The latter feels like speaking to a human, the former more like your trying to get your voice commands into a state machine.
Around 1998 I broke my collarbone and had to use Dragon Dictate.
I found that for general subjects it was quite difficult to use because of the fairly poor recognition rate.
But when I talked about computers, it got almost everything right. I assumed it must have been trained by the developers, who talked about computers mostly.
This is another special purpose vocabulary, so it seems as if it would have a good chance of a high recognition rate.
The voice interface experience(in general) so far is like trying to make a really stupid person do something for you. Out of context misunderstandings are the worst because it breaks your flow trying to understand why that happens and how to fix it.
I imagine that voice to code would be like standing over the shoulders of a junior coder who knows the syntax and some techniques just enough to follow orders but has no idea whats doing and when gets it wrong will be very wrong.
I think there's a difference between communicating your intent to a machine, which is hopeless since it has no model of intention; and commanding a machine to reproduce something.
Ie., when you're managing your house you want something that can be communicated in an infinite number of ways, but the "AI" accepts a tiny finitude of ways.
However when programming it seems like we arent asking the machine to "write a function to do X", but rather saying, "def open-paren star args...."
This seems like a pretty trivial problem to solve.
> However when programming it seems like we arent asking the machine to "write a function to do X", but rather saying, "def open-paren star args...."
Click the link first and take a look at what is being showcased, because your comment is the exact opposite of what they demo when you visit the HN link.
You're right... So, yes, it will be largely useless (as shown) for actual programming.
But I suspect there'll be a subset of its features consistent with my comment that will be actually useful.
Programming, via Naur/Ryle, is always a kind of theory building. And unless you're basically copy/pasting, it's a novel theory of some area (, business process, etc.).
That's something where intentions arent even really communicable as such, since the art of programming is sketching possible theories as a way of finding out what we ought intend.
So this is another gimmik with maybe marginal improvements at the edges.
There may well be examples of this, but while the blind developers I have known (a small sample, I admit) typically use screen-reader technologies to navigate and read code, they use a keyboard to write and edit it.
I don't disagree with that. I just meant I don't think it's going to have mainstream appeal. A wheel chair also makes a disabled person super productive if the alternative is not being able to go anywhere at all, but it doesn't make wheelchairs super appealing to people with healthy two legs if you see what I mean.
I am happy to take a severe deficit over not being able to work at all. When my back was acting up, I could not physically use my left side. Dictation was the ONLY way I could code. By the end of this period, my output was back up to 95% of my typed output - especially as I don’t type code nearly as fast as I do general language writing.
"Writing is thinking. To write well is to think clearly. That’s why it’s so hard." ~David McCullough
This not only holds for literature but also for programming. Concerning the hard part, I would argue that is the reason why it is not called "talking is thinking".
"If you're thinking without writing, you only think you are thinking." -Leslie Lamport
Even though now speech recognition rate is really high, but I wonder how many authors use speech to write articles. The comparison may make sense. And I think there's few.
I don't use voice assistants any more due to privacy concerns but I wrote some similar software in 2010s. I'm fluent in English, but with the current tech, the success rate for me giving commands to a machine is still 50/50.
> I could never be productive programming like this.
It's likely to work much better than a generic speech-to-text model due to fine-tuning.
Plus, consciously or not, we will adapt our human language to the English-ML "pidgin" (e.g. by introducing a more efficient grammatical structures, using a specific subset of vocabulary).
The way I see it is that it's not much different from giving commands to your dog, writing a Google query, writing a Stable Diffusion prompt. It'll get better. Manual input is not as fast as speech though and that's where I see the issue.
This looks to be much more heavily using GPT3/Codex/Copilot, which I've found to be eerily effective. It basically feels like a voice interface to Copilot. The main difference between these and something like Google Home is how effectively they pick up on context. "Hey Github" would be able to use all the code in the file as context, so when you say "wrap this in a function", it'll have an idea of what you mean, without that function having to be explicitly programmed. Voice assistants have to _always_ be in a voice space, so context is very limited. And generally the way Google home-style voice assistants are created is by programming specific actions linked to specific phrases. ML helps make the phrase matching flexible, but the action is usually entirely explicitly coded. Using Codex would let the action be ML influenced as well.
If Copilot is any indicator of effectiveness, then I have high hopes for this! I've always wanted to program while stationary biking :)
I think yes this could be a real multiplier for seniors, you're doing something you have done lots of times before just a bit different you know pretty much everything you need to do, describe it until it is in a state where you can through and finish it off. Exactly like a stationary bike or out in the garden with your kid type thing.
IF the voice analysis was any good of course. But maybe it will also be able to be better than typical voice analysis because the syntax is limited, when programming I use a much more limited vocabulary than when writing literary criticism. So while text to speech is total crap for handling complex literary phrasing it might be adequate for programming structures.
I think this is great for:
a) people who are visually impaired or have issues with their hands/fingers
b) people who aren't programmers; if you could make it more Scratch-like then this is amazing tool for showing off power of programming
I think the biggest use case for this is accessibility. There are plenty of people who permanently or temporarily cannot use a keyboard (and/or mouse). This will be great for those users.
For the average dev, I agree this is more of a novelty.
> GitHub seems to be more high-level. It figures out the syntax and what you actually want to write.
To me, looks like it's feeding your voice input to Copilot that then generates the code output just as before. So, the same strength and weaknesses of Copilot apply (and you can probably mimic it locally with a voice input method you control, just dictate comments for copilot)
I am highly suspicious of new tech coming in the guise of 'accessibility'. As someone goin blind, a lot of things toted as good for me are cumbersome and bad.
Maybe this will be different, and that'd be neat. Though I just think more expressions of code is neat. I also know the accessibility you're talkin about isn't for blindness.
That being said I can talk about code decently well, but if you've never heard code come out of text-to-speech, well, it's painful.
I bring up the text-to-speech because if speech is input, it would make sense for speech to also be the output. Selfishly, getting a lot of developers to spend time coding through voice might end up with some novel and well thought out solutions.
I do think there will be big advancements in the text-to-speech realm. I've noticed some ML projects imitating voices surprisingly well and while it's not quite there yet - it's already a bit less grating than it was even a few years ago.
For sight problems you are correct. But voice input is valuable by itself. I had chronic tendonitis in my wrists a few years ago. I looked into voice coding and it was difficult to set up. Fortunately for me I've been able to adapt with a vertical mouse and split keyboard.
As a new dad, I would love to have the voice-to-text accuracy and speed I get on my Pixel phone on my desktop OS. Done right, I could easily see myself using it more often than when I have my youngling in one arm as I've been WFH for the better part of the last 6 years of work.
For repetitive tasks like preparing a report in the demo, saying is definitely faster than typing. It's quite impressive if your boss ask you to prepare one and the report is done in less than two minutes.
However, I too really doubt if there's any better use cases than simple tasks, let alone everyone would hear what you ask the AI to do in the office. Oh my! How embarrassing am I?
I've tried writing documentation and fiction using text-to-speech and, for me, it doesn't work because the apparently the of my part brain I use to think about what I'm going to say is the same part I use to actually say it, so I can't do both things at once. I end up writing far more slowly than I can type.
In case anyone else stopped after watching the video, if you scroll down a bit further you see the list of
FEATURES
Write/edit code
Just state your intent in natural language and let Hey, GitHub! do the heavy lifting of suggesting a code snippet. And if you don't like what was generated, ask for a change in plain English.
Go to the next method
Code navigation
No more using mouse and arrow keys. Ask Hey, GitHub! to...
go to line 34
go to method X
go to next block
Control the IDE
"Toggle zen mode", “run the program”, or use any other VisualStudio Code command.
Code Summarization
Don’t know what a piece of code does? No problem! Ask Hey, GitHub! to explain lines 3-10 and get a summary of what the code does.
249 comments
[ 3.9 ms ] story [ 259 ms ] threadI have been playing with using Whisper + Github Copilot in Vim [0]. The Whisper text transcription runs offline with a custom C/C++ inference and I use Copilot through the copilot.nvim plugin for Neovim. The results were very satisfying.
Edit: And just in case there is interest in this, the code is available [1]. It would be very awesome if someone helps to wrap this functionality in a proper Vim plugin.
[0] https://youtu.be/3flN9kTcZJY
[1] https://github.com/ggerganov/whisper.cpp/tree/master/example...
The problem with current voice programming systems is they're just too slow so I end up getting impatient and using my fingers anyway
Could it also do a virtual keyboard, but a custom layout to not trigger arm, elbow and hand pains?
VERY PROMISING, in any case you can just manually fill the gaps with the keyboard!
Why?
The problem with speech to code has always been that precise syntax is hard, but AI codegen solves that.
So, no, it might not take off, but I feel like if it does, then it means ai-codegen will become the dominant way code is crafted.
That would be paradigm shifting.
It’s inconceivable that it wouldn’t be.
Easier isn’t always better.
I don't want to disparage their work, because it's really impressive, but "fill null values of column Fare with average column values" is closer to AppleScript than it is to natural language.
It solves the issue of trying to speak obscure code syntax like “close parenthesis semicolon newline”.
That’s enough to lower the barrier to entry for many people; I don’t know how good it is practically but it’s disingenuous to suggest it’s not offering a novel solution to an old problem.
The biggest problem is that talking sucks. You presumably can handle voice input as well as is possible, yet here we are typing to you anyway, and for good reason. Even if the natural language part is nailed, you may as well type in that natural language.
I imagine it will bring some quality of life improvements to those with certain disabilities, but I don't see why the typical developer would want to go in that direction.
Usually this kind of exploratory work involves a lot of Googling and copy-pasting snippets from Stackoverflow without putting too much time in trying to deeply understand things. If you get out what you want - great, if not, back to Google.
I would totally enjoy being able to tell my IDE to "call foo with bar and string hello there end string with a block of gee times two" or something, instead of:
Just that, not having to think about typing different symbols would be a serious quality of life feature for me.Poland ditched a similar QWERTZ-based layout in favour of this: https://pl.wikipedia.org/wiki/Plik:Polish_programmer%27s_lay...
It's basically the standard US layout but the right alt (AltGr) is a modifier. So, for example, AltGr+A gives "ą".
I don't see why something similar can't be done for the Czech alphabet.
It probably could — we already can't fit all the letters with diacritics on the number row, so "ď, ť, ň, ó" are key combos. But as far as I know, Czech uses diacritics a bit more than Polish (e.g. for sounds that are digraphs in Polish), consider:
"Že se nestydÍŠ, nutit lidi psÁt ČeskÉ speciÁlnÍ znaky pomocÍ dvojhmatŮ!" — that's 10 modifiers just for the diacritics.
Having ALL diacritics as modifier combos would make typing actual texts even more annoying than programming is now.
Because in my experience it is very often like "Call Peter" -> "Today it's sunny in NY".
Are either of those companies investing particularly heavily into voice agents? Certainly neither of them has anywhere near the kind of power of something like Copilot.
Also, a general agent is way different from one that's specific to writing code.
On macOS it still seems pretty good - I have carpal tunnel syndrome and by Thursday or Friday most weeks I end up using Siri to dictate not code but a lot of conversations in Slack, pull requests, iMessage, etc. In fact, I wrote this reply with Siri right now.
Now it's barely worth attempting, because it gets it wrong more than it gets it right.
Code is much more constrained by language syntax though.
Even for the "call peter" example, while the input is easy, the expected range of inputs that Siri should handle and be able to differentiate it from is huge.
Of course this is still a problem for e.g. defining variable names, where you could say anything.
But yeah, something about talking to a device which gets things wrong all the time is ridiculously distracting, at least for me.
Sometimes I look back at the road after trying to workout what it interpreted and I feel scared how focused on the phone I became.
This means it's an assistive technology, but hardly "a generational paradigm change in how to write code".
If machines were amazing at Speech-to-Text, okay, sure. But while the capabilities are impressive, they still kinda suck at it.
I don't see how text to code would be faster than typing. And even if it is, typing speed is not really a limiting factor in the speed at which I can produce code.
But yeah, that is pretty close to amazing.
I kinda forgot about it after seeing that the Rhasspy community experimented with it, and it had issues with short utterances and a slow startup time.
Yet, it hasn't stuck. I'm exclusively using Siri to set timers. Most people are like me, or don't use it at all. Some use assistants for googling factoids or something. Fidelity wise, it's really underwhelming.
It's not a social acceptance issue, because people would still use it at home, and they don't. It's a small chance there's some key UI insight missing (discoverability for one), but I doubt it. Even with perfect UI, natural language is quite flawed when you're dealing with technical details (see exhibit on variable naming).
Anyway, the chances of Github solving this in an exceptionally difficult subdomain, as a side project, seems like a... Let's say, long shot.
That said, the silver lining in all these billions spent on voice interfaces is accessibility. For some people, these things are a life saver.
The hard part will be open plan offices.
It’s bad enough that so many meetings are now zoom/teams and proximity to coworkers means you end up hearing their side of their meetings.
Just wait until all the devs are coding this way too.
https://www.youtube.com/watch?v=ILfTrUreS00
"USER!! UNDERSCORE LIMIT!! EQUALS TWO THOUSAND AND FORTY EIGHT!"
Why?
Can't really see myself working like this in an office, plane, cafe, with music on (my favorite way to code), in the house where my partner is also working. Then as others have said, editing might suck.
If it was a neural link then I'd be in agreement.
https://www.theverge.com/2022/11/8/23446821/microsoft-openai...
I can see this helping as an accessibility tool, but beyond that I don’t think it will be useful. This kind of assumes you know everything about what you’re doing, most of the time you don’t.
“Go to line 35” “Open the model controller” “Show the get method and set method side by side”
So all the more power to them, but I am very skeptical. Especially since co-pilot has zero knowledge of the formal semantics of programming languages.
This is a lot different than the half ass auto complete that it already does since that at least has some context.
That sounds exhausting when we have spent countless human years developing languages which let us communicate our intentions precisely to computers.
If you don't do this there is no ambiguity detector. Meaning it's entirely possible for the computer to interpret what you are saying completely different than intended, yet it is a perfectly valid interpretation. So the only one who can qualify if it got it right is you.
What is another bonus; if you generated function transfromAtoB(a: A) above, then you only have to do:
And it'll come up with the reverse.It's not hard to write yourself, but it's boring and error prone (some of these structures are huge). Now I press tab a bunch of times, and run the tests to see if it worked. I am also not that worried i'm infringing someone's open source code; this is all way to custom to look like anything else. That's where this shines; things where it verbatim copies something, you should've been using a library anyway.
Statically typing and using typescript definitely works better than other combinations I have tried (C# was pretty bad last I tried it, JS is good but often subtly wrong because of type issues).
But for a glimpse of the future watch The Expanse or read William Gibson's Agency.
It does look like we've made some progress in the 15 years since. I do wonder how this would work in an office setting though - so much noise, so much distraction, and so much crosstalk between programmers...
Everyone gets a throat mic and the cubicle farm is full of unintelligible whispering instead of clacking of keyboards? Can't wait for the future. /s
:/
Then it seemed to just die off. I don't think it was bad technology, because I don't think novelty value was enough to account for its popularity - you had to put hours in to get it to work well, it wasn't a casual toy.
What's changed since then in terms of technology? Unless it's very significant, I suspect it will go the same way. Apart from an assistive technology viewpoint, my gut instinct is that it's not that satisfying or rewarding talking to a computer all day.
Can I have transcription that can then turn my rambling into neat and concise prose?
The story for the most mainstream-popular dictation softwares is kind of funny. Back in the late 90's there was Dragon NaturallySpeaking and IBM's ViaVoice. In early 00s, after a financial fraud and bankruptcy involving both the then current Dragon owners and Goldman Sachs, they got bought by Scansoft. Scansoft bought Nuance, began to use its name, and then got exclusive rights for ViaVoice (!) from IBM.
Now, in March this year, Nuance has been acquire by Microsoft.
Training data is now abundant compared to twenty years ago, and so is computation power. That means training can be much more complex now.
The underlying technology is now typically neural networks (broadly speaking), whereas twenty years ago it might have been Hidden Markov Models.
Overall, recognition quality, even without speaker-specific training, is now on a very different level than back then. Whether it’s considered good is a matter of opinion. But it’s significantly better than twenty years ago.
It's hugely significant - look at this graph of Google's speech model accuracy across 2013 to 2017:
https://sonix.ai/packs/media/images/corp/articles/history-of...
Or this that shows a similar pattern:
https://cdn.static-economist.com/sites/default/files/externa...
1) Improvements in speech to text, as others have mentioned
2) Improvements in language models (and model size) allowing for more flexible interpretation of speech. This isn't dictation anymore. It's more like instruction. You don't have to tell the computer exactly what to write, you tell it in much more broader terms. Eg "pull this out into a function". Or "delete the cookie before creating the transaction". Or "lint the file".
That's my guess anyways! This mostly feels like a voice interface to Copilot in a lot of ways. Can't say whether it'll be effective, but I'd love to be able to program while I'm e.g. on a stationary bike!
"insert curly brace", "insert semicolon", "insert insertion", etc. does not sound to fun.
https://www.youtube.com/watch?v=YKuRkGkf5HU
The demos are in Ruby, but I could imagine that languages with strong type-aware auto-completion could be easier to do.
For that reason I think this will be less appealing to developers than GitHub may think, otherwise I think it's a cool idea.
I found that for general subjects it was quite difficult to use because of the fairly poor recognition rate.
But when I talked about computers, it got almost everything right. I assumed it must have been trained by the developers, who talked about computers mostly.
This is another special purpose vocabulary, so it seems as if it would have a good chance of a high recognition rate.
I imagine that voice to code would be like standing over the shoulders of a junior coder who knows the syntax and some techniques just enough to follow orders but has no idea whats doing and when gets it wrong will be very wrong.
Ie., when you're managing your house you want something that can be communicated in an infinite number of ways, but the "AI" accepts a tiny finitude of ways.
However when programming it seems like we arent asking the machine to "write a function to do X", but rather saying, "def open-paren star args...."
This seems like a pretty trivial problem to solve.
Click the link first and take a look at what is being showcased, because your comment is the exact opposite of what they demo when you visit the HN link.
But I suspect there'll be a subset of its features consistent with my comment that will be actually useful.
Programming, via Naur/Ryle, is always a kind of theory building. And unless you're basically copy/pasting, it's a novel theory of some area (, business process, etc.).
That's something where intentions arent even really communicable as such, since the art of programming is sketching possible theories as a way of finding out what we ought intend.
So this is another gimmik with maybe marginal improvements at the edges.
This not only holds for literature but also for programming. Concerning the hard part, I would argue that is the reason why it is not called "talking is thinking".
Even though now speech recognition rate is really high, but I wonder how many authors use speech to write articles. The comparison may make sense. And I think there's few.
> I could never be productive programming like this.
It's likely to work much better than a generic speech-to-text model due to fine-tuning.
Plus, consciously or not, we will adapt our human language to the English-ML "pidgin" (e.g. by introducing a more efficient grammatical structures, using a specific subset of vocabulary).
The way I see it is that it's not much different from giving commands to your dog, writing a Google query, writing a Stable Diffusion prompt. It'll get better. Manual input is not as fast as speech though and that's where I see the issue.
If Copilot is any indicator of effectiveness, then I have high hopes for this! I've always wanted to program while stationary biking :)
IF the voice analysis was any good of course. But maybe it will also be able to be better than typical voice analysis because the syntax is limited, when programming I use a much more limited vocabulary than when writing literary criticism. So while text to speech is total crap for handling complex literary phrasing it might be adequate for programming structures.
For the average dev, I agree this is more of a novelty.
I bet if we use our imaginations, we’ll think of a lot of places were using voice to code could come in handy.
Personally, I’ve been waiting for it for a few decades.
The creator of TCL has RSI and has been using voice since the late 1990’s
https://web.stanford.edu/~ouster/cgi-bin/wrist.php
Thought we were really close 10 years ago when Tavis Rudd developed a system:
https://youtu.be/8SkdfdXWYaI
GitHub seems to be more high-level. It figures out the syntax and what you actually want to write.
This would help if you barely knew the language.
Time to learn Rust or Scala with a little help from machine learning.
This statement probably didn't happen. The closest thing to it was 10 years after the quote is usually supposed to have happened and was about a single model of a single machine: https://geekhistory.com/content/urban-legend-i-think-there-w...
To me, looks like it's feeding your voice input to Copilot that then generates the code output just as before. So, the same strength and weaknesses of Copilot apply (and you can probably mimic it locally with a voice input method you control, just dictate comments for copilot)
Maybe this will be different, and that'd be neat. Though I just think more expressions of code is neat. I also know the accessibility you're talkin about isn't for blindness.
That being said I can talk about code decently well, but if you've never heard code come out of text-to-speech, well, it's painful.
I bring up the text-to-speech because if speech is input, it would make sense for speech to also be the output. Selfishly, getting a lot of developers to spend time coding through voice might end up with some novel and well thought out solutions.
However, I too really doubt if there's any better use cases than simple tasks, let alone everyone would hear what you ask the AI to do in the office. Oh my! How embarrassing am I?
FEATURES
Write/edit code
Just state your intent in natural language and let Hey, GitHub! do the heavy lifting of suggesting a code snippet. And if you don't like what was generated, ask for a change in plain English. Go to the next method
Code navigation
No more using mouse and arrow keys. Ask Hey, GitHub! to...
Control the IDE"Toggle zen mode", “run the program”, or use any other VisualStudio Code command.
Code Summarization Don’t know what a piece of code does? No problem! Ask Hey, GitHub! to explain lines 3-10 and get a summary of what the code does.
Explain lines 3 - 10