" But to understand this dataset’s significance, we must first ... "
understand the terms of the license under which your contributions will be governed and applied, your ability to know where this is being deployed, your rights to be paid for skilled work.. (?)
Unless that work is greater than the work of manually migrating the codebase, I doubt it's 'more' work. And if that is the case, the whole endeavour is pointless.
1) it’s new code that works well in 90-95% of the cases, but fails in unexpected ways and cannot be debugged, you have to generate a new piece of code that works in the above failure but has its own: works 90-95% times and fails in mysterious ways
OR
2) it looks like code, it reads like code but humans can’t figure out what it does
OR
3) it is a collection of blocks of code put in order automatically to do a certain job, that works in 90-95% of cases and…
Even assuming that there is a specification, which is already a tall order:
- if this is something like automated theorem provers, it's something like (2). It's correct, provably correct even, but it's a bunch of nonsense that provides no insight to humans.
- if this is something like natural language query generators, it's something like (1), but with a much lower success rate (closer to 50% IME).
I don't even need to specify that AI-generated code is interesting, but we aren't even remotely close to being there.
A lot of IDE's and editors already support autocomplete, intellisense or something similar. I think AI generated code is never going to work but implemented as a autocomplete it can be helpful. I am using https://www.tabnine.com/ for vs code now and that works nice but its very limited The thing that i'm mostly missing are configuration for the aggression and length of the provided options.
I'm not saying it will never work, never is a long time, but it's not the perfect domain for AI. It's kind of the opposite, actually.
First and most importantly, programming computers is a very precise endeavor. The logic needs to be exactly correct, not only statistically correct. 'Close enough' won't cut it, not even remotely, and not even for relatively unimportant software.
Second, the general problem is undecidable. This isn't a roadblock per se, because we are reasonably good at other undecidable problems (viz. garbage collection), but it means simple algorithmical approaches won't work.
Third, software is in a weird place because it requires working at different abstraction levels simultaneously. Often, top-level specifications are fuzzy and incomplete, but some parts require absolute precision and we need to "drop down" to a lower level of abstraction. Humans are able to make the process work (kinda) using lots of common sense, something machines are currently very bad at.
If you require the operator of the 'AI' to fill in the details, you just invented a very complicated compiler.
Finally, rarely if ever present-day software is made once and never changed: the output needs to be inspectable and maintainable, other software might need to call into it, etc.
If the pipeline is more complicated than the software itself, I might as well be writing the code myself.
I can see some minor, specific tasks being increasingly done with AI, and I can see tools making more and more use of AI technology, but AI generated code isn't even on the horizon.
It seems like many of those issues would be solved by having a human write a comprehensive test suite, and then the AI would write an implementation that can pass all of the tests.
Eventually you could also have another separate AI that learns to generate the test suite itself from instructions given to it by a human (or even another AI!).
Most program implementations would not be perfect, but as we know, software written by humans certainly isn’t perfect either.
Are you sure of that? Microsoft has recently introduced so called AI into VisualStudio, and IMO, it's idiocy rather than intelligence.
If you don't know what I mean and you do conveniently have a VS license, try turn the AI shit one and you will soon see it yourselves.
I'm using an old version of codium. But thanks for the recommendation. I will check it out when I ever have to use VisualStudio again or when it gets added to vs code.
IBM is an awful company to deal with. We were interested in getting an IBM data provider for .Net Core to connect to DB2 database. Figuring out the cost was insanely hard and nobody could answer which version we needed. I was then provided with a trial version, which did not work for version of .Net Core I was using. I had to dig deep and spend days figuring out why it wasn't working. I was put in touch with IBM engineer and figured out the issue before he did. After they provided the correct version, the setup was extremely annoying. Placing license files in a folder. Then, I had issues with errors and IBM deleted a bunch of forum threads and I was presented with many dead links.
Many questions on IBM forum were answered with "I sent you a PM", followed by people screaming "why can't you just openly share the solution".
Documentation was awful to non-existent.
They just want to nickle and dime everyone.
I decided IBM doesn't deserve any money, and I am using good ole OleDb to connect to DB2.
“I am angry with this company for their past transgressions. I don’t want their new venture to succeed so I’ll bad mouth them. To make it seem relevant to the thread, I’ll spin this anecdote as emblematic of a culture that can’t succeed.”
I remember them hyping blockchain in their tv ads a few years ago -- how they were going to revolutionize everything with it. Pure hype.
IBM is a great old brand, but all things pass, and it's way overdue for them. New things grow where old things die, and I'm ager to see what follows them.
Interesting. We've been running an IBM server on prem for 15 years and have had maybe 4 hours downtime. It saved our bacon when ransomware hit as the only thing that didn't get trashed. It holds our PII. It is expensive, and we were considering switching away until the ransomware incident.
I am really interested in applicability of models trained on this specific dataset. For sure I know that if I'd encounter an "algorithmic competition" style code in a PR I'd hardly ever accept it.
What I really need is some system which has as much information as I have about the world & be able to learn things I throw at it. I don't even need it to invent calculus. Just learn what I teach or what books or media can teach then ask doubts & go about it's tasks. Whether it is software or hardware as well I don't care
This seems like a way for IBM to hype up technology modernization services maybe more than any other possible scenario to me. I don’t believe this will yield useful code generation that will replace real software developers.
I can see it now, IBM low code / no code platform for migrating COBOL on the IBM Mainframe to really bad javascript on the IBM Cloud.
I agree - it is a big compiler that has backward compatibility for java apps. The problem is code logic where at one point we had to use stringIO to read every byte - now we have fileIO to read a record. Old code worked but was too slow, Marcos and calls gave us screenIO in bytes 40by60 today 1024by720 seems small - the old code is just too slow - COBOL/IMS is in service today but 45 magabytes a second, means batch runs all night and why there's COBOL/DB2 - In sum without c++, information engineering and the dropping of recycled ideas this goes nowhere .. better is to build new faster chips, bit processing tools(ML), Such that I can say build me a light saber(NL) and there is a call to 3D printers, metal fabrication, laser delimiters, and butane assembly!
TO HAVE ON DEMAND MANUFACTURING of goods Eg(star trek replication)
No seriously, AI will take years to understand the context.
Coding is more about capturing real world knowledge into a functional program. It involves so much learning about how to translate processes, workflow and regulations into a binary that it seems naive to have an AI code generation tool we even can't understand.
Lifting my head from the CodeNet launch and joining the discussion here.
the 14M code samples, spread over dozens of languages, for over 4000 tasks that each solve a well specified problem in english. Each problem has certain constraints. Each task also is annotated with very unique metadata of a test set, which the solution must pass to be a successful solution. This meets the functional spec. Now, on to the constraints that follow.. After meeting the functional spec, the solution must meet a runtime constraint, and a memory constraint as well. All this metadata of what that solution achieved is carefully annotated for each of the 14M samples.
Now, what could this be used for:
1. Since there are functional and non-functional codes for each problem that is precisely defined, data could be used for AI model to learn how to debug and modify the code to make it "correct" - the test data will be critical here for model's learning to be reinforced and driven with.
2. Improve the performance of the code - similar to above, since we know for each solution what the performance was, for a given problem, learn the ways to improve performance.
3. similar to 2 above, improve memory
4. code similarity, since we can compare the underlying graph as well, which are in the metadata for many samples, and AST generators are provided too..
5. Code translation since solutions for the same problem are in polyglot of languages, code translation is another critical usecase.
27 comments
[ 3.4 ms ] story [ 76.2 ms ] threadunderstand the terms of the license under which your contributions will be governed and applied, your ability to know where this is being deployed, your rights to be paid for skilled work.. (?)
How would that effect the job market for human software engineers?
1) it’s new code that works well in 90-95% of the cases, but fails in unexpected ways and cannot be debugged, you have to generate a new piece of code that works in the above failure but has its own: works 90-95% times and fails in mysterious ways
OR
2) it looks like code, it reads like code but humans can’t figure out what it does
OR
3) it is a collection of blocks of code put in order automatically to do a certain job, that works in 90-95% of cases and…
OR
4) ??
- if this is something like automated theorem provers, it's something like (2). It's correct, provably correct even, but it's a bunch of nonsense that provides no insight to humans.
- if this is something like natural language query generators, it's something like (1), but with a much lower success rate (closer to 50% IME).
I don't even need to specify that AI-generated code is interesting, but we aren't even remotely close to being there.
First and most importantly, programming computers is a very precise endeavor. The logic needs to be exactly correct, not only statistically correct. 'Close enough' won't cut it, not even remotely, and not even for relatively unimportant software.
Second, the general problem is undecidable. This isn't a roadblock per se, because we are reasonably good at other undecidable problems (viz. garbage collection), but it means simple algorithmical approaches won't work.
Third, software is in a weird place because it requires working at different abstraction levels simultaneously. Often, top-level specifications are fuzzy and incomplete, but some parts require absolute precision and we need to "drop down" to a lower level of abstraction. Humans are able to make the process work (kinda) using lots of common sense, something machines are currently very bad at. If you require the operator of the 'AI' to fill in the details, you just invented a very complicated compiler.
Finally, rarely if ever present-day software is made once and never changed: the output needs to be inspectable and maintainable, other software might need to call into it, etc. If the pipeline is more complicated than the software itself, I might as well be writing the code myself.
I can see some minor, specific tasks being increasingly done with AI, and I can see tools making more and more use of AI technology, but AI generated code isn't even on the horizon.
Eventually you could also have another separate AI that learns to generate the test suite itself from instructions given to it by a human (or even another AI!).
Most program implementations would not be perfect, but as we know, software written by humans certainly isn’t perfect either.
IBM is an awful company to deal with. We were interested in getting an IBM data provider for .Net Core to connect to DB2 database. Figuring out the cost was insanely hard and nobody could answer which version we needed. I was then provided with a trial version, which did not work for version of .Net Core I was using. I had to dig deep and spend days figuring out why it wasn't working. I was put in touch with IBM engineer and figured out the issue before he did. After they provided the correct version, the setup was extremely annoying. Placing license files in a folder. Then, I had issues with errors and IBM deleted a bunch of forum threads and I was presented with many dead links.
Many questions on IBM forum were answered with "I sent you a PM", followed by people screaming "why can't you just openly share the solution".
Documentation was awful to non-existent.
They just want to nickle and dime everyone.
I decided IBM doesn't deserve any money, and I am using good ole OleDb to connect to DB2.
IBM is a great old brand, but all things pass, and it's way overdue for them. New things grow where old things die, and I'm ager to see what follows them.
I can see it now, IBM low code / no code platform for migrating COBOL on the IBM Mainframe to really bad javascript on the IBM Cloud.
No seriously, AI will take years to understand the context.
Coding is more about capturing real world knowledge into a functional program. It involves so much learning about how to translate processes, workflow and regulations into a binary that it seems naive to have an AI code generation tool we even can't understand.
Now, what could this be used for: 1. Since there are functional and non-functional codes for each problem that is precisely defined, data could be used for AI model to learn how to debug and modify the code to make it "correct" - the test data will be critical here for model's learning to be reinforced and driven with. 2. Improve the performance of the code - similar to above, since we know for each solution what the performance was, for a given problem, learn the ways to improve performance. 3. similar to 2 above, improve memory 4. code similarity, since we can compare the underlying graph as well, which are in the metadata for many samples, and AST generators are provided too.. 5. Code translation since solutions for the same problem are in polyglot of languages, code translation is another critical usecase.