ChatGPT: The end of new programming languages?
She noted that if a new programming language is created, it will have far less training data for ChatGPT than old programming languages do. So, to the degree that ChatGPT or similar AIs become an important part of writing code, the new language will be at a disadvantage even if it's intrinsically superior. So people won't use it, so it will never catch up to the existing ones in ChatGPT support due to lack of training data. So people won't use it, so...
In response, I noted that something like SudoLang is rooted in ChatGPT and so doesn't seem to have that problem. But I also said that her point seemed valid and worth thinking about. It seems like the outcome may be that the next truly popular programming language may be intrinsically tied to an AI. It won't necessarily be SudoLang or much like it, but that's not necessarily the only way it could be done.
50 comments
[ 3.6 ms ] story [ 117 ms ] threadOn the bright side, this is not a new phenomenon. The number of "things" that a new programming language needs to provide that are outside of the language itself has been growing for a long time now. A new language already isn't going to find purchase if it doesn't have a package manager, a build system, a hosted package repository, a standardized pretty printer, a test runner, a documentation generator, a StackOverflow presence, an active community on both synchronous and asynchronous communication platforms, and IDE support. Needing a trained LLM "just" raises the bar for IDE support. However, unlike all of the other "things", this is something that can't really leverage prior art or standardized protocols.
Of course, even to this day there are some people who program without IDEs at all. In order to bootstrap their own LLM training data, new languages will need to fiercely compete for this ever-shrinking segment.
I'm not sure what I agree with, but I think it's interesting to think about.
My off the top of my head guess is it makes it easier to make new programming languages and have them be usable by people, but it reduces the value of new programming languages. So both things happen. We end up with more new programming languages because they're easier to make and use, but more of the usage concentrates on the existing languages and the ones that ChatGPT is best at. So both a bigger long tail, but more concentrated 'big winners'
(If this sounds dangerous to you -- to use a technology that humans aren't smart enough to understand ourselves -- I guess I'd agree.)
You can give it a BNF grammar for the target language of choice, if you like. Not only is C/C++ the "new assembly," but every other high-level programming language can be thought of that way as well.
It replied chirpily that it could do that, in so many words, and output a grammar (which I didn't test) that solely parsed MySQL identifiers rather than the double-quoted kind. It wasn't precisely an invalid response, but it was clearly inappropriate for a human-like reply and showed what I consider a qualitative absence of consciousness.
I don't know why, but this seems to be a pattern with SQL - I have never asked it about MySQL, and indeed MySQL results tend to pollute my Google searches, but there may just not be enough other training material.
It's starting to aggravate me every time someone concedes it only is useful sometimes, and you have to check it, because it is starting to seem like unwarranted "praising with faint damns".
The reason I say this is because with something with fewer examples but still with docs available online, specifically, AutoHotKey; ChatGPT's not nearly as good.
We'll get to where you're going, but we're not there quite just yet.
OP is saying that without this initial bootstrap corpus, this won’t work well and hence people accustomed to this won’t adopt the language.
> the new language will be at a disadvantage even if it's intrinsically superior.
Languages are not usually compared like this.
No one seriously designing new languages is thinking "gee better not do that because of large language models prior training".
Even amongst existing languages, the level training among large language models isn't a deciding factor in writing production code more so than if the language is suitable for the problem domain.
From my own experience, writing software today doesn't that much different than it was 20 years ago using some java IDE from that era (Borland? maybe I'm remembering it better than it was). I can imagine what development will be like next year with increasing integration of copilot like tools, but I can't imagine with any confidence what software development will be like 10 years from now.
Something more interesting to think about is what would a programming language designed FOR llms to use look like?
When web browsing was enabled it was able to learn these by itself.
But if training is used, I think you would want to define a full self-contained platform. That way you know the LLM knows everything it needs to translate the user's requirements into code.
A group like DeepMind might go as far as to integrate code execution into their system somehow. Maybe it's like an LLM but trained to iteratively write code for some self-contained platform.
Or there could be a full platform designed so the specification fits well-within a context window.
Something like K or oK is very compact. You could give it a library to save data and display some UI stuff and define the whole thing in a context window maybe.
Using natural language, it is incredibly difficult to make unambiguous statements. For most of the twentieth century, the problem was regarded as so severe that analytic philosophers would encode their most important points in symbolic logic. And it's not like we have one symbolic logic here; there are literally dozens of higher-order logics, each with their own syntax, and some of which are mutually incompatible.
Meanwhile, complex, difficult-to-master conlangs such as Ikthuil and Lojban have flourished in pockets of the philosopherse simply because they provide, essentially, the kind of clarity that any basic programming language provides.
I can easily imagine AI/ML inertia, as OP wonders about, and I can also easily imagine forms of e.g. Lojban, Ikthuil, or a mishmash of several higher-order symbolic logics becoming the basis for a new ML-informed programming language, but if your imagination has you talking to the computer the way you'd talk to a subordinate, in English or some other natural language, then do I have a monkey's paw for you.
If you don't think that's ever going to be possible with AI (notice how I used the word "eventually" in my post) then I don't know what to say to you.
It's like the joke:
The programmer's wife said: "Please go to the store and buy a carton of milk and if they have eggs, get a dozen."
He came back with 12 cartons of milk.
She said, "why did you buy twelve cartons of milk?"
He responds "they had eggs"
(there are rare features like this in some parsers, like dangling else, but they are considered design problems)
Language does not handle most business process nor efficiency goal and even security restrictions. People and logic do. Little of that language (!) is in the written word. It is done mostly by failures and historic-obvious-cultural practice.
Emulation does not mean 'comprehension' of all the purposes involved in craftsmanship of handling data for business or for even computation.
Most AI-solved problems are search reduction for spoon fed problem areas of data.
Some random things that seem difficult to get an AI to deal with:
1) Things that people expect you to know by observation, but not be told and not to say.
2) Things that people expect you to know/adhere to, in spite of being told something different.
3) Strategically creating the impression that you do or do not have the ability to change your course of action.
Punctuation problem.
I tried to trick it into running an infinite loop by giving it an obfuscated recursion for "generating squares" and overriding the exception that stops it.
It informed me that the action would infinite loop (spotted it better than me) and suggested that I didn't need such a complicated structure to calculate squares. Then it proposed a tessaract formula that "is bound to the laws of physics".
This is a problem that humans struggle with too.
The two ways you typically end up getting around this problem with a programming language; a subset of natural language that uses very short and terse sentences with a simple and limited grammar;
... or extremely laborious clarifications on the exact sense in which everything is meant, as seen in legal text which needs to be immune sophistry and various adversarial (mis)interpretations. e.g.I knew the AI hype machine was real but I didn’t realise it had replaced decades of computing security already.
That said, it doesn't seem to be outside the realm of possibility. Which is why I said it.
ChatGPT does not understand the hard parts of Rust I read somewhere (no evidence).
But in the longer term it’s more likely that programming will morph into something much more inherently linked to AI.
People said computers and computer games would be the death of board games, but instead board games have exploded in popularity, I think mainly cause it’s easier to create fabulous board games with computers.
New technologies tend to be overestimated in the short term and underestimated in the long term.
But, ultimately, new products, programming languages included, succeed because they satisfy needs not serviced by the existing incumbents. And I don’t see that changing.
Right now we’ve got our low level stuff, machine code where you’re telling the CPU what to do. Then C etc: want to read a file? You’re manipulating pointers, reading bytes etc etc. Then we have higher level languages where we just call readFile(). The standard library “knows” what we want to do when we say “read file” via the code contained within it.
I can imagine AI enabling an extra layer. We’ll say “format this string as a US telephone number” and it’ll make “XXX-XXX-XXXX” out of it. But it isn’t going to anywhere near as exciting as the current hype wave makes it out to be because if we want to get unambiguous responses that don’t make unwanted assumptions our prompts are going to have to be really, really specific. The most efficient way of doing that will probably be via some kind of structured prompt language. So… another programming language. It might look very different from the languages we use today (or it might not!) but it’ll be a programming language all the same.
I think most devs have a hard time grasping how revolutionary the spreadsheet is for non programmers for accounting, etc tasks. Yes they get abused, but its a huge powertool that solves a lot of tasks that previously required teams of programmers.
Looking back, we might wonder how spreadsheets impacted programming languages?
Still, the # of software devs is huge and increasing, and there are plenty of new indie projects made by people who are passionate about new languages and/or want to do things more fun and possibly faster rather than more reliable. Take Zig, a very early language which competes directly with Rust, C, D, Go, and more; but there are real projects being developed with Zig (e.g. Bun). These projects and the developers using niche languages are growing a community which others can use in the future, and writing code which an LLM can use; and while the communities aren’t growing nearly as fast as ones of larger languages, there’s probably an exponential dropoff of community/code growth + usability growth. More importantly, even if these niche languages never become popular (more likely than not), they’ll still be used by a passionate group who make projects 2x slower (since they have less resources/LLMs), but would otherwise be unmotivated and not make anything at all.
I can't imagine every programming practitioner is going to switch to AI tools even if they get a lot better than they are now. There will be contrarian people making new languages and writing in them. Some of them might catch on and become popular.
Also, it seems that the AI tools don't suck that bad even for made-up toy languages. Maybe it'll struggle if the new language is using lots of unconventional ideas.
ChatGPT will not innovate for you. If I'm a researcher for new programming languages, I can't imagine it would be all that useful for me in the research part. Maybe in the implementation part. I certainly wouldn't stop research just because AI is suddenly a thing.
Also, it seems like finetuning is pretty cheap, I expect it might not be very difficult to add knowledge about a new programming language to an existing foundational model made for code. The tools to do this will get better and more accessible.
I think my hotter take is that new programming languages that are really nice for humans are perhaps also really nice for the AIs. So LLMs being a thing wouldn't necessarily change how you design programming languages.
And my even more hot take is that I think the impact of these tools is waaaaay overestimated for now.
If some absolutely, obviously amazing programming language is suddenly invented that is only usable with AI assistance that could change things. I am not holding my breath though.