A language is LLM-optimized if there’s a huge amount of high-quality prior art, and if the language tooling itself can help the LLM iterate and catch errors
The Validation Locality piece is very interesting and really got my brain going. Would be cool to denote test conditions in line with definitions. Would get gross for a human, but could work for an LLM with consistent delimiters. Something like (pseudo code):
```
fn foo(name::"Bob"|genName(2)):
if len(name) < 3
Err("Name too short!")
print("Hello ", name)
return::"Hello Bob"|Err
```
Right off the bat I don't like that it relies on accurately remembering list indexes to keep track of tests (something you brought up), but it was fun to think about this and I'll continue to do so. To avoid the counting issue you could provide tools like "runTest(number)", "getTotalTests", etc.
I get that this is essentially vibe coding a language, but it still seems lazy to me. He just asked the language model zero-shot to design a language unprompted. You could at least use the Rosetta code examples and ask it to identify design patterns for a new language.
> Humans don't have to read or write or undestand it. The goal is to let an LLM express its intent as token-efficiently as possible.
Maybe in the future, humans don't have to verify the spelling, logic or grounding truth either in programs because we all have to give up and assume that the LLM knows everything. /s
Sometimes, I read these blogs from vibe-coders that have become completely complacent with LLM slop, I have to continue to remind others why regulations exist.
Imagine if LLMs should become fully autonomous pilots on commercial planes or planes optimized for AI control and the humans just board the plane and fly for the vibes, maybe call it "Vibe Airlines".
Why didn't anyone think of that great idea? Also completely remove the human from the loop as well?
An LLM is optimized for its training data, not for newly built formats or abstractions. I don’t understand why we keep building so-called "LLM-optimized" X or Y. It’s the same story we’ve seen before with TOON.
If a new programming language doesn’t need to be written by humans (though should ideally still be readable for auditing), I hope people research languages that support formal methods and model checking tools. Formal methods have a reputation for being too hard or not scaling, but now we have LLMs that can write that code.
llm-optimized in reality would mean you asked and answered millions of stackoverflow questions about it and then waited a year or so for all the major models to retrain.
I'm looking for a language optimized for use with coding agents. Something which helps me to make a precise specification, and helps the agent meet all the specified requirements.
so where are the millions of line code you need to train the LLM in your new language? Remember AI is just a statistic prediction thing. No input -> No output
I think I've come full circle back to the idea that a human should write the high-level code unassisted, and the LLM should autocomplete the glue code and draft the function implementations. The important part is that the human maintains these narrow boundaries and success criteria within them. The better the scaffolding, the better the result.
Nothing else really seems to make sense or work all that well.
On the one extreme you have people wanting the AI to write all the code on vibes. On the other extreme you have people who want agents that hide all low-level details behind plain english except the tool calls. To me these are basically the same crappy result where we hide the code the wrong way.
I feel like what we really need is templating instead of vibes or agent frameworks. Put another way, I just want the code folding in my editor to magically write the code for me when I unfold. I just want to distribute that template and let the user run it in a sandbox. If we're going to hide code from the user at least it's not a crazy mess behind the scenes and the user can judge what it actually does when the template is written in a "literate code" style.
The primary constraint is the size of the language specification. Any new programming language starts out not being in the training data, so in context learning is all you've got. That makes it similar to a compression competition - the size of the codec is considered to be a part of the output size in such contests, so you have to balance codec code against how effective it is. You can't win by making a gigantic compressor that produces a tiny output.
To me that suggests starting from a base of an existing language and using an iterative tree-based agent exploration. It's a super expensive technique and I'm not sure the ROI is worth it, but that's how you'd do it. You don't want to create a new language from scratch.
I don't think focusing on tokenization makes sense. The more you drift from the tokenization of the training text the harder it will be for the model to work, just like with a human (and that's what the author finds). At best you might get small savings by asking it to write in something like Chinese, but the GPT-4/5 token vocabularies already have a lot of programming related tokens like ".self", ".Iter", "-server" and so on. So trying to make something look shorter to a human can easily be counter productive.
A better approach is to look at where models struggle and try to optimize a pre-existing language for those issues. It might all be rendered obsolete by a better model released tomorrow of course, but what I see is problems like this:
1. Models often want to emit imports or fully qualified names into the middle of code, because they can't go backwards and edit what they already emitted to add an import line at the top. So a better language for an LLM would be one that doesn't require you to move the cursor upwards as you type, e.g. Python/JS benefits here because you can run an import statement anywhere, languages like Java or Kotlin are just about workable because you can write out names in full and importing something is just a convenience, but languages that force you to import types only at the very top of the file are going to be hell for an LLM.
Taking this principle further it may be useful to have a PL that lets you emit "delete last block" type tokens (smarter ^H). If the model emits code that it then realizes was wrong, it no longer has to commit to it and build on it anyway, it can wipe it and redo it. I've often noticed GPT-5 use "no op" patterns when it emits patches, where it deletes a line and then immediately re-adds the exact same line, and I think it's because it changed what it wanted to do half way through emitting a patch but had no way to stop except by doing a no-op.
The nice thing about this idea is that it's robust to model changes. For as long as we use auto-regression this will be a problem. Maybe diffusion LLMs find it easier but we don't use those today.
2. As the article notes, models can struggle with counting indentation especially when emitting patches. That suggests NOT using a whitespace sensitive language like Python. I keep hearing that Python is the "language of AI" but objectively models do sometimes still make mistakes with indentation. In a brace based language this isn't a problem, you can just mechanically reformat any file that the LLM edits after it's done. In a whitespace sensitive language that's not an option.
3. Heavy use of optional type inference. Types communicate lots of context in a small number of tokens, but demanding the model actually write out types is also inefficient (it knows in its activations what the types are meant to be). So what you want is to encourage the model to rely heavily on type inference even if the surrounding code is explicitl, then use a CLI tool that automatically adds in missing type annotations, i.e. you enrich the input and shrink the output. TypeScript, Kotlin etc - all good for this. Languages like Clojur...
But it reminds me of the SEO guys optimizing for search engines. At the end of the day, the real long term strategy is to just "make good content", or in this case, "make a good language".
In the futuristic :) long term, in "post programming-language world" I predict each big llm provider will have its own propertiary compiler/vm/runtime. Why basically do transpiling if you can own the experience and result 100% and compete on that with other llm providers.
The real question is what existing language is perfect for LLMs? Is Lisp? ASM? We know some LLMs are better at some languages but what existing language are they best at? Would be interesting to see. I know one spot they all fail at is niche programming libraries. They have to pull down docs or review raw code pulled down for the dependency, issue is in some languages like C# those dependencies are precompiled to bytecode, Java too.
edit but also see cons[3] - maybe viable for very constrained domains, with strict namespace management and handling drop into debugger. Also, after thinking more, it likely only sounds nice (python vs lisp training corpus and library ecosystems; and there's mcp-py3repl (no reflection but otherwise more viable), PAL, etc.) Still - curious.
In theory (I've seen people discuss similar things before though), homoiconicity and persistent REPL could provide benefits - code introspection (and code is a traversable AST), wider persistent context but in a tree structure where it can choose breadth vs depth of context loading, progressive tool building, DSL building for given domain, and (I know this is a bit hype vibe) overall building up toolkit for augmented self-expanding symbolic reasoning tools for given domain / problem / etc. (starting with "build up toolkit for answering basic math questions including long sequences of small digits where you would normally trip up due to your token prediction based LLM mechanism"[2]). Worth running some quick experiments maybe, hm :)
P.S. and thinking of agentic loops (a very uh contemporary topic these days), exposing ways to manage and construct agent trees and loops itself is (while very possibly recipe for disaster; either way would need namespaces not to clash) certainly captivating to me (again given effective code/data traversal and modification options; ideally with memoization / caching / etc.)
[3] cons (heh): hallucination in the metaprogramming layer and LLMs being fundamentally statistical models and not well trained for Lisp-like langs, and inevitable state pollution (unless some kind of clever additional harness applied) likely removes much of the hype...
22 comments
[ 2.5 ms ] story [ 42.6 ms ] thread``` fn foo(name::"Bob"|genName(2)): if len(name) < 3 Err("Name too short!")
```Right off the bat I don't like that it relies on accurately remembering list indexes to keep track of tests (something you brought up), but it was fun to think about this and I'll continue to do so. To avoid the counting issue you could provide tools like "runTest(number)", "getTotalTests", etc.
One issue: The Loom spec link is broken.
Maybe in the future, humans don't have to verify the spelling, logic or grounding truth either in programs because we all have to give up and assume that the LLM knows everything. /s
Sometimes, I read these blogs from vibe-coders that have become completely complacent with LLM slop, I have to continue to remind others why regulations exist.
Imagine if LLMs should become fully autonomous pilots on commercial planes or planes optimized for AI control and the humans just board the plane and fly for the vibes, maybe call it "Vibe Airlines".
Why didn't anyone think of that great idea? Also completely remove the human from the loop as well?
Good idea isn't it?
https://martin.kleppmann.com/2025/12/08/ai-formal-verificati...
Nothing else really seems to make sense or work all that well.
On the one extreme you have people wanting the AI to write all the code on vibes. On the other extreme you have people who want agents that hide all low-level details behind plain english except the tool calls. To me these are basically the same crappy result where we hide the code the wrong way.
I feel like what we really need is templating instead of vibes or agent frameworks. Put another way, I just want the code folding in my editor to magically write the code for me when I unfold. I just want to distribute that template and let the user run it in a sandbox. If we're going to hide code from the user at least it's not a crazy mess behind the scenes and the user can judge what it actually does when the template is written in a "literate code" style.
The primary constraint is the size of the language specification. Any new programming language starts out not being in the training data, so in context learning is all you've got. That makes it similar to a compression competition - the size of the codec is considered to be a part of the output size in such contests, so you have to balance codec code against how effective it is. You can't win by making a gigantic compressor that produces a tiny output.
To me that suggests starting from a base of an existing language and using an iterative tree-based agent exploration. It's a super expensive technique and I'm not sure the ROI is worth it, but that's how you'd do it. You don't want to create a new language from scratch.
I don't think focusing on tokenization makes sense. The more you drift from the tokenization of the training text the harder it will be for the model to work, just like with a human (and that's what the author finds). At best you might get small savings by asking it to write in something like Chinese, but the GPT-4/5 token vocabularies already have a lot of programming related tokens like ".self", ".Iter", "-server" and so on. So trying to make something look shorter to a human can easily be counter productive.
A better approach is to look at where models struggle and try to optimize a pre-existing language for those issues. It might all be rendered obsolete by a better model released tomorrow of course, but what I see is problems like this:
1. Models often want to emit imports or fully qualified names into the middle of code, because they can't go backwards and edit what they already emitted to add an import line at the top. So a better language for an LLM would be one that doesn't require you to move the cursor upwards as you type, e.g. Python/JS benefits here because you can run an import statement anywhere, languages like Java or Kotlin are just about workable because you can write out names in full and importing something is just a convenience, but languages that force you to import types only at the very top of the file are going to be hell for an LLM.
Taking this principle further it may be useful to have a PL that lets you emit "delete last block" type tokens (smarter ^H). If the model emits code that it then realizes was wrong, it no longer has to commit to it and build on it anyway, it can wipe it and redo it. I've often noticed GPT-5 use "no op" patterns when it emits patches, where it deletes a line and then immediately re-adds the exact same line, and I think it's because it changed what it wanted to do half way through emitting a patch but had no way to stop except by doing a no-op.
The nice thing about this idea is that it's robust to model changes. For as long as we use auto-regression this will be a problem. Maybe diffusion LLMs find it easier but we don't use those today.
2. As the article notes, models can struggle with counting indentation especially when emitting patches. That suggests NOT using a whitespace sensitive language like Python. I keep hearing that Python is the "language of AI" but objectively models do sometimes still make mistakes with indentation. In a brace based language this isn't a problem, you can just mechanically reformat any file that the LLM edits after it's done. In a whitespace sensitive language that's not an option.
3. Heavy use of optional type inference. Types communicate lots of context in a small number of tokens, but demanding the model actually write out types is also inefficient (it knows in its activations what the types are meant to be). So what you want is to encourage the model to rely heavily on type inference even if the surrounding code is explicitl, then use a CLI tool that automatically adds in missing type annotations, i.e. you enrich the input and shrink the output. TypeScript, Kotlin etc - all good for this. Languages like Clojur...
But it reminds me of the SEO guys optimizing for search engines. At the end of the day, the real long term strategy is to just "make good content", or in this case, "make a good language".
In the futuristic :) long term, in "post programming-language world" I predict each big llm provider will have its own propertiary compiler/vm/runtime. Why basically do transpiling if you can own the experience and result 100% and compete on that with other llm providers.
The functions are optionally tested using formal verification. I plan to enable this by default soon, as time allows.
These functions that get written can then be composed, and enzymes that run in the cloud actively look for functions to fuse.
Also more people use it, the faster the compiler gets via network scaling laws.
It's very much research at the moment, but kinda works.
Jupyter Notebook style interface with the beginnings of some image and media support.
https://prometheus.entrained.ai
Can try looking at some of the examples or trying something yourself.
Would love some feedback.
> From Tool Calling to Symbolic Thinking: LLMs in a Persistent Lisp Metaprogramming Loop
https://arxiv.org/abs/2506.10021
edit but also see cons[3] - maybe viable for very constrained domains, with strict namespace management and handling drop into debugger. Also, after thinking more, it likely only sounds nice (python vs lisp training corpus and library ecosystems; and there's mcp-py3repl (no reflection but otherwise more viable), PAL, etc.) Still - curious.
In theory (I've seen people discuss similar things before though), homoiconicity and persistent REPL could provide benefits - code introspection (and code is a traversable AST), wider persistent context but in a tree structure where it can choose breadth vs depth of context loading, progressive tool building, DSL building for given domain, and (I know this is a bit hype vibe) overall building up toolkit for augmented self-expanding symbolic reasoning tools for given domain / problem / etc. (starting with "build up toolkit for answering basic math questions including long sequences of small digits where you would normally trip up due to your token prediction based LLM mechanism"[2]). Worth running some quick experiments maybe, hm :)
P.S. and thinking of agentic loops (a very uh contemporary topic these days), exposing ways to manage and construct agent trees and loops itself is (while very possibly recipe for disaster; either way would need namespaces not to clash) certainly captivating to me (again given effective code/data traversal and modification options; ideally with memoization / caching / etc.)
[1] https://arxiv.org/abs/2506.10021
[2] https://www.youtube.com/watch?v=AWqvBdqCAAE on need for hybrid systems
[3] cons (heh): hallucination in the metaprogramming layer and LLMs being fundamentally statistical models and not well trained for Lisp-like langs, and inevitable state pollution (unless some kind of clever additional harness applied) likely removes much of the hype...
Sort of like Rosetta code but for AI and more complex tasks?