Please post these in a form that doesn't require jumping through complex hoops to watch or read. If that means waiting until they're uploaded to a more accessible place, that's fine. On HN there's no harm in waiting. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Did you watch the video? The AI is interpreting the comments as instructions on what code to generate. That's 95% of the solution, since the comments are just english and there already exists an abundance of NLU models in things like Alexa, Google, etc, that take speech input and produce english output, like the code comments.
I tried signing in with my Microsoft account as well, nope, they want you to definitely go ahead and fill out a registration form for Build conference https://register.build.microsoft.com/, not gonna happen. Hope they learn not to paywall conferences of this kind, their competition just puts it out on YouTube live.
It includes a segment with Sam Altman doing python code generation from nothing other than signatures and comment strings. Pretty incredible -- assuming the demo isn't entirely smoke and mirrors.
If people are truly novice, with zero programming experience, how would they know the code is correct? If not, how to debug it?
I would say this is more promising for scenarios to generate more formularitive things like business report generation. But even that, it requires in depth understanding of what those data/tables really means, and how to handle exceptions, etc.
When AI can reliably convert business-speak into efficient bug-free code, everyone will be out of a job, because the business owners will ask it to write them another AI to replace every other task their business does.
Kind of. A theme in programming since the beginning has been automation. You're using a computer anyway to do a more or less well defined task, naturally programming itself is one of the prime targets for automation.
Programming languages are automation tools. Libraries. Frameworks. It should be pretty clear this is a lasting trend, and doesn't necessarily mean programmers will have more or less jobs (due to well known effects of automation such as enabling new applications and increasing demand from increased productivity). It does mean you probably need to keep learning to stay relevant, and use those tools to your advantage!
It's not obvious that it works until you hit the play button and it starts at the right time. Seems like the only way to get it to play automatically is to embed it in a tweet like this: https://twitter.com/modeless/status/1263222139840167936
It's Sam Altman demoing a massive Open AI model that was trained on GitHub OSS repos using a Microsoft supercomputer. It's not Intellicode, but the host says that they're working on compressing the models to a size that could be feasible in Intellicode. The code model uses English-language comments, or simply function signatures, to generate entire functions. Pretty cool.
Wow, this has the ability to be a total gamechanger. You have to be really observant about the bugs though, I would have totally missed the one with the price discount without executing it.
Similar problem to automated driving - as long as it's better than most humans, occasional bugs will be ok. Virtually no software is bug free.
It's much more difficult problem than automated driving though - for software, the space of intents of the user is orders of magnitude greater in size. It's the job of the model to determine the intent of the "programmer". Perhaps we could meet the model half way and come up with heavily-structured natural language to communicate intent.
Programming skills are directly correlated to programmer ability to debug, I will go as far stating; programming is not about writing code, but in ability to find bugs and figure out how to resolve them.
I've always found it easier to debug code I wrote, mostly because I find it the easier to read the code I wrote, since I understand the author's intent.
But as the programmer finds bugs and corrects them, they are simultaneously generating more, accurate data for training. So over time this should theoretically improve
I notice that the bug was in the user's failure to communicate the intent of the scalar. Presumably with regular use users would learn to be more clear and/or anticipate the likely fixes to ambiguous labels.
Also, since it would be used to build tests as well, I'd expect such misunderstandings to be pretty obvious. I would be willing to bet you'd see a net reduction in bugs, and a substantial reduction in typo related bugs.
But if you mean by lowered barrier of entry you mean the population of programmers would be less competent, yes bugs in the design might increase, however being able to more quickly get to the point of evaluating a design is a great way to learn better design.
Yeah, I guess developers will have to all become data scientists to help train these A.I's to write better code :-) Perhaps there will be a new business model around selling your higher quality code to help train the A.I to be better and better... so we need to label code "good" and "crap" so the A.I. can avoid learning from crappy code :-)
It’s not that we don’t have enough code it’s that the code doesn’t do what we want.
To get this you can just grab any random 18 year old who knows js and have them hack something out. No one hires 18 year old js hackers though and there’s a reason for that.
I don't think programmers will become obsolete. They will just waste less time on boilerplate and reading Stackoverflow articles to figure out how to do XYZ. Why not have an AI do that for you so that you can focus on the creative stuff? Programming tools help us work more productively, which leads to larger and more complex systems in less time - a win for everyone.
I've worked with developers from all around the globe. While it's true that some cannot even write fizzbuzz, some others can be extremely brilliant individuals with an excellent work ethic.
This is really cool. However, I doubt it can write more than very simple functions. That may be enough to be useful however. It would be nice if they created a demo page where we could try this out. This use case is a little different than the auto-complete one.
Can this freaky A.I. also generate the corresponding unit tests?
Or, for TDD, generate the unit tests first based on the function name and description. Then, if the dev updates any of those tests, or adds more tests, use that information in auto generating the appropriate code.
Towards the end of that section, he mentions they have also used it to generate unit tests. I doubt it's doing full TDD, but it seems they are part of the way there.
How does this do compared to other models? Is this a totally cutting edge result? On the surface, it seems quite impressive, but sans an environment to try it out with, I cannot be entirely sure. Still, this does make me question whether I chose a safe career, haha.
The thing is, I'd really need to see a live demo to see how good this is. Making mistakes is actually kind of a big issue; as most people know, debugging code is harder than writing it. And a lot of the language models which can write impressive-seeming text also generate masses of garbage. There's no way to know whether this was cherrypicked or not.
The mere fact that it can extract meaning from text like this is already really impressive though.
I've read a fair number of papers on neural program synthesis lately. To me, these seemed to be obviously cherry picked examples, so you can't really evaluate the whole system based on them.
However, this is fairly impressive for a couple reasons. First, the system constructs programs from natural language descriptions, rather than examples of input-output pairs or a formal specification, which are the most common settings for program synthesis. Second, they're generating full blown python, not a smaller, domain specific language.
Finally, and this is pretty mind-blowing, is the seamless, idiomatic use of loops, branches, and function calls. I haven't seen previous program synthesis tools able to generate such complex code. They're typically limited to simple linear programs with less than about 100 lines. Complex control flow and function calls are still beyond their reach for the most part.
I'm not an active researcher in neural program synthesis, so my statements may not reflect the current state of the art.
I honestly thought that the most promising route forward for program synthesis would be a model that incorporated knowledge of the syntax and semantics of code. Most likely, a model that manipulated, or at least had some view of, the program's AST. This seems to be just throwing a giant Transformer model at github.
Fine tuning a vanilla language model on a giant corpus of code feels like a dead end for the field, long-term. It seems obvious to me that humans are doing something more than just statistical pattern recognition and generation when we write and reason about code.
Then again, it's hard to argue with results. I'm sure lots of pre-neural network voice recognition researchers were in love with the elegance of their hidden markov models.
Edit: Also, everyone should go try the FlashFill feature in Microsoft excel. As far as I know, it's the only example of program synthesis shipped in a consumer facing production system, and it works shockingly well.
> Fine tuning a vanilla language model on a giant corpus of code feels like a dead end for the field, long-term. It seems obvious to me that humans are doing something more than just statistical pattern recognition and generation when we write and reason about code.
Yeah, this is the main reason why I would be interested in more examples. But, if this thing was trained on all of GitHub, I could imagine that it come up with decent-looking code for a lot of examples; a beefy, smarter Google with some rudimentary contextual understanding, if you will. Still, the presence of any mistakes is a no-go and I'd be really interested how it reacts to more realistic, specific requirements.
But yeah, I'd figure a model for code generation would have to have some kind of knowledge of syntax and semantics, rather than doing pure statistical pattern matching, to be of any real use. It would not only have to generate, but also to debug its code (I wonder whether you could do that purely with statistical pattern recognition). I might be wrong, of course, but I would be surprised if that is enough to write complex code.
Calling the field "statistical pattern matching" might be underselling it a bit, even if technically accurate on some level. I mean, syntax/semantics are clearly not the problem, those are the easiest to learn (see the paper above). If anything, I'm scared of it writing syntactically correct nonsense (or even worse, subtly-flawed-but-correct-looking code).
I am also hedging my hopes of this working on "more realistic" scenarios. It does produce code that looks natural to us, but i expect it to show clear "seams" where its understanding of something isn't deep enough.
But maybe this is just a question of how much compute (and network size/"depth") you invest. On a certain level we're also just some recurrent LSTM :)
>> Edit: Also, everyone should go try the FlashFill feature in Microsoft excel. As far as I know, it's the only example of program synthesis shipped in a consumer facing production system, and it works shockingly well.
And it's not a giant language model trained on a gigantic dataset. Rather, if memory serves, it's a buch of task-specific DSLs and rules, all hand-written from scratch.
I don't know how FlashFill works in 2020,
but from [1] I learn that the original implementation was a brute-force enumeration (with clever heuristics along the lines of CDCL (= conflict-driven clause learning in SAT solvers) for speeding up common cases) of a small DSL for string manipulation. This was (and still is)
the state-of-the-art approach to programming-by-example
program synthesis.
That's a nice, formal way of putting it, thank you :)
(Sorry I really should have refreshed my memory on Gulwani et al. I think I've even linked the paper on an HN comment before.)
Oh, btw, I doubt they're doing this with a language model nowadays. Unless FlashFill has suddendly started filling cells for email addresses with haikus etc...
Ha. You hit the nail on the head. There is no rigorous way to measure AI-generated anything. (to my knowledge) So every demo is "ooh look at this" and actual performance is not scientifically evaluated, because we don't know how. This includes images, text, etc.
I can't even imagine what it's like to have so much money that you can spend time working on things like this which are so incredibly unlikely to ever become useful. Congrats and I hope you guys discover a great product some day.
But there is the thing, the natural description of a function is not always this unambiguous.
When you are telling a function to 'compute XYZ', what you are actually doing is 'check whether X.a exists, if so execute branch 1), else branch 2)'.
If the logic gets really complicated, then describing it accurately in human language isn't necessarily faster than doing it in code directly. Otherwise, we don't need invent programming languages like at all, we can just write compilers to interpret and execute human languages.
And I am interested, as whether the model itself is conditioned on the type constraint of class. It is neat that they pick Python in this case. But if it is Java or other static typed language, would this system condition its generation not only the natural text, but also the resulted type system? My bet, per my understanding of the language modeling approach they use is, they are not doing this, due to very high complexity and cost of the training, and domain adaptation.
Overall, this again is an interesting demo. But I think for code generation based on human language to be useful, we are really in a scenario, that you need to go 99% accurate for it to be remotely practical.
But that would require the condition on the type system, meaning the code-gen needs to understand the object's interface, which while not impossible in current techniques, but hard enough due computation complexity.
Again I don't dispute this tool being interesting. But claims it to be ground breaking or game changing is simply not right.
Majority of programmers time, is not typing down the code. It is to look at the comment/description, think about it, edit some code, then rethink then edit again.
This tool has potential to solve some typing time, but it still not going to things fundamentally.
This might be more useful for a task like "read files off a list, and download them in parallel, with no more than 20 concurrent downloads." That particular task might be a one-liner in some programming languages, but there are a lot of programs like that which need significant bookkeeping and/or boilerplate even though their plain-language description of intended behavior is not complicated.
Or implementing a sophisticated protocol that has a formal specification. If you can express the correct behavior in some kind of pithy pseudocode, a tool like this could "compile" that to code in various programming languages. Like a super-powered version of SWIG.
I assume you have read a few scientific papers before.
This tool is the programming equivalent of an AI writing scientific papers based on an abstract. It can follow all the formalities really well. It can write beautiful English sentences. It might write formulas or produce graphs. It will dot the i's and cross the t's.
But it's unclear whether what it says is actually correct and logically coherent when used in the main part of the paper (and not just for the introduction of the paper) or just pleasing-sounding nonsense.
It can certainly help make engineers look more productive, just like it could help someone write papers at record speed. Whether the results can/will have any deeper value is yet to be determined. Maybe it will just be used for the "boring" tasks - like the paper introduction.
My personal fear is that it will be very good at writing code that looks ok, even though there is a serious flaw. Essentially, programmers tend to become good at spotting irregularities in the code corresponding to common human errors. The mistakes of this AI might be much harder to spot because they don't stand out in the same way.
The samples are pretty impressive, and that's coming from someone who's intimately familiar with the internals of these these kind of models.
From bitter experience, though, I also know how unreliable these models can be. It's possible (indeed, likely) they generated 20 samples, threw away 19 that were garbage and just showed you the one that looked nice.
If - and that's a big if - it reliably generates good-quality code, then it would probably be a nice productivity boost (~25%, particularly when it comes to tests).
Based on what I've seen over the past few years, though, I'm skeptical.
I'am a bit confused, is this built by OpenAI or Microsoft?
Microsoft released the paper IntelliCode Compose: Code Generation Using Transformer [1] 4 days ago and there is no attribution to anyone from OpenAI in it.
Are those two entirely separate and yet exactly similar initiatives?
I am aware of this, I am referring to the video, where Sam Altman (CEO of OpenAI) is presenting the demo and saying "we have built", while Kevin Scott (CTO of MSFT) is saying that it's the first time he has seen that. So this is clearly marketed as OpenAI's work, not just saying that the model is based on their work.
157 comments
[ 0.22 ms ] story [ 210 ms ] threadAlso, please don't rewrite titles to make them baity. That's against the site guidelines: https://news.ycombinator.com/newsguidelines.html
Edit: other users have helpfully posted a link to the video that anyone can view, so we've switched to that from https://mybuild.microsoft.com/sessions/6c6ecd46-c39c-49d8-ba... and restored the submission.
Build me a class which computes the larger of two integers.
The AI is smart enough to write it.
If people are truly novice, with zero programming experience, how would they know the code is correct? If not, how to debug it?
I would say this is more promising for scenarios to generate more formularitive things like business report generation. But even that, it requires in depth understanding of what those data/tables really means, and how to handle exceptions, etc.
Programming languages are automation tools. Libraries. Frameworks. It should be pretty clear this is a lasting trend, and doesn't necessarily mean programmers will have more or less jobs (due to well known effects of automation such as enabling new applications and increasing demand from increased productivity). It does mean you probably need to keep learning to stay relevant, and use those tools to your advantage!
If anyone knows a way to link to the start of the demo at 28m30s, or thereabouts, we can modify it again. (Edit: maybe https://blog.twitter.com/en_us/topics/product/2018/video-tim... can be used to make that work?)
It's not obvious that it works until you hit the play button and it starts at the right time. Seems like the only way to get it to play automatically is to embed it in a tweet like this: https://twitter.com/modeless/status/1263222139840167936
You can view the demo at https://twitter.com/i/broadcasts/1OyKAYWPRrWKb starting around 29:00.
It's Sam Altman demoing a massive Open AI model that was trained on GitHub OSS repos using a Microsoft supercomputer. It's not Intellicode, but the host says that they're working on compressing the models to a size that could be feasible in Intellicode. The code model uses English-language comments, or simply function signatures, to generate entire functions. Pretty cool.
It's much more difficult problem than automated driving though - for software, the space of intents of the user is orders of magnitude greater in size. It's the job of the model to determine the intent of the "programmer". Perhaps we could meet the model half way and come up with heavily-structured natural language to communicate intent.
Also, since it would be used to build tests as well, I'd expect such misunderstandings to be pretty obvious. I would be willing to bet you'd see a net reduction in bugs, and a substantial reduction in typo related bugs.
But if you mean by lowered barrier of entry you mean the population of programmers would be less competent, yes bugs in the design might increase, however being able to more quickly get to the point of evaluating a design is a great way to learn better design.
So the developer's role will shift to:
1) writing good enough descriptions of the code to be generated by the AI model
2) fixing any little issues in the generated code
To get this you can just grab any random 18 year old who knows js and have them hack something out. No one hires 18 year old js hackers though and there’s a reason for that.
Or, for TDD, generate the unit tests first based on the function name and description. Then, if the dev updates any of those tests, or adds more tests, use that information in auto generating the appropriate code.
The thing is, I'd really need to see a live demo to see how good this is. Making mistakes is actually kind of a big issue; as most people know, debugging code is harder than writing it. And a lot of the language models which can write impressive-seeming text also generate masses of garbage. There's no way to know whether this was cherrypicked or not.
The mere fact that it can extract meaning from text like this is already really impressive though.
However, this is fairly impressive for a couple reasons. First, the system constructs programs from natural language descriptions, rather than examples of input-output pairs or a formal specification, which are the most common settings for program synthesis. Second, they're generating full blown python, not a smaller, domain specific language.
Finally, and this is pretty mind-blowing, is the seamless, idiomatic use of loops, branches, and function calls. I haven't seen previous program synthesis tools able to generate such complex code. They're typically limited to simple linear programs with less than about 100 lines. Complex control flow and function calls are still beyond their reach for the most part.
I'm not an active researcher in neural program synthesis, so my statements may not reflect the current state of the art.
I honestly thought that the most promising route forward for program synthesis would be a model that incorporated knowledge of the syntax and semantics of code. Most likely, a model that manipulated, or at least had some view of, the program's AST. This seems to be just throwing a giant Transformer model at github.
Fine tuning a vanilla language model on a giant corpus of code feels like a dead end for the field, long-term. It seems obvious to me that humans are doing something more than just statistical pattern recognition and generation when we write and reason about code.
Then again, it's hard to argue with results. I'm sure lots of pre-neural network voice recognition researchers were in love with the elegance of their hidden markov models.
Edit: Also, everyone should go try the FlashFill feature in Microsoft excel. As far as I know, it's the only example of program synthesis shipped in a consumer facing production system, and it works shockingly well.
Yeah, this is the main reason why I would be interested in more examples. But, if this thing was trained on all of GitHub, I could imagine that it come up with decent-looking code for a lot of examples; a beefy, smarter Google with some rudimentary contextual understanding, if you will. Still, the presence of any mistakes is a no-go and I'd be really interested how it reacts to more realistic, specific requirements.
But yeah, I'd figure a model for code generation would have to have some kind of knowledge of syntax and semantics, rather than doing pure statistical pattern matching, to be of any real use. It would not only have to generate, but also to debug its code (I wonder whether you could do that purely with statistical pattern recognition). I might be wrong, of course, but I would be surprised if that is enough to write complex code.
Calling the field "statistical pattern matching" might be underselling it a bit, even if technically accurate on some level. I mean, syntax/semantics are clearly not the problem, those are the easiest to learn (see the paper above). If anything, I'm scared of it writing syntactically correct nonsense (or even worse, subtly-flawed-but-correct-looking code).
But maybe this is just a question of how much compute (and network size/"depth") you invest. On a certain level we're also just some recurrent LSTM :)
And it's not a giant language model trained on a gigantic dataset. Rather, if memory serves, it's a buch of task-specific DSLs and rules, all hand-written from scratch.
[1] O. Polozov, S. Gulwani, FlashMeta: A Framework for Inductive Program Synthesis. https://www.microsoft.com/en-us/research/wp-content/uploads/...
(Sorry I really should have refreshed my memory on Gulwani et al. I think I've even linked the paper on an HN comment before.)
Oh, btw, I doubt they're doing this with a language model nowadays. Unless FlashFill has suddendly started filling cells for email addresses with haikus etc...
Want to bet on that?
But there is the thing, the natural description of a function is not always this unambiguous.
When you are telling a function to 'compute XYZ', what you are actually doing is 'check whether X.a exists, if so execute branch 1), else branch 2)'.
If the logic gets really complicated, then describing it accurately in human language isn't necessarily faster than doing it in code directly. Otherwise, we don't need invent programming languages like at all, we can just write compilers to interpret and execute human languages.
And I am interested, as whether the model itself is conditioned on the type constraint of class. It is neat that they pick Python in this case. But if it is Java or other static typed language, would this system condition its generation not only the natural text, but also the resulted type system? My bet, per my understanding of the language modeling approach they use is, they are not doing this, due to very high complexity and cost of the training, and domain adaptation.
Overall, this again is an interesting demo. But I think for code generation based on human language to be useful, we are really in a scenario, that you need to go 99% accurate for it to be remotely practical.
But I think the example given of unit testing - ie. natural language description of specific behavior of function -> code is extremely useful.
But that would require the condition on the type system, meaning the code-gen needs to understand the object's interface, which while not impossible in current techniques, but hard enough due computation complexity.
Again I don't dispute this tool being interesting. But claims it to be ground breaking or game changing is simply not right.
Majority of programmers time, is not typing down the code. It is to look at the comment/description, think about it, edit some code, then rethink then edit again.
This tool has potential to solve some typing time, but it still not going to things fundamentally.
Or implementing a sophisticated protocol that has a formal specification. If you can express the correct behavior in some kind of pithy pseudocode, a tool like this could "compile" that to code in various programming languages. Like a super-powered version of SWIG.
In all seriousness, the demo really looks amazing. I'm curious to see more elaborate, real world examples though.
This tool is the programming equivalent of an AI writing scientific papers based on an abstract. It can follow all the formalities really well. It can write beautiful English sentences. It might write formulas or produce graphs. It will dot the i's and cross the t's.
But it's unclear whether what it says is actually correct and logically coherent when used in the main part of the paper (and not just for the introduction of the paper) or just pleasing-sounding nonsense.
It can certainly help make engineers look more productive, just like it could help someone write papers at record speed. Whether the results can/will have any deeper value is yet to be determined. Maybe it will just be used for the "boring" tasks - like the paper introduction.
My personal fear is that it will be very good at writing code that looks ok, even though there is a serious flaw. Essentially, programmers tend to become good at spotting irregularities in the code corresponding to common human errors. The mistakes of this AI might be much harder to spot because they don't stand out in the same way.
From bitter experience, though, I also know how unreliable these models can be. It's possible (indeed, likely) they generated 20 samples, threw away 19 that were garbage and just showed you the one that looked nice.
If - and that's a big if - it reliably generates good-quality code, then it would probably be a nice productivity boost (~25%, particularly when it comes to tests).
Based on what I've seen over the past few years, though, I'm skeptical.
Are those two entirely separate and yet exactly similar initiatives?
[1]: https://arxiv.org/abs/2005.08025v1
GPT-2 is built by OpenAI