I agree. What is missing from the discussion about DSLs are the importance of tooling such as linters, LSPs, etc, to give the LLMs further context.
For example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.
The gap I've hit generating GPU kernels with agents
code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator.
dsl stays small is doing all the heavy lifting here
the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (specially if its not json/yaml or just function chain based). as the number of things that can mix and match increase you are basically looking at a whole system prompt just describing the new language.
this brings us to the second part. step 2: after dsl is 'planned' (note they use the java compiler), the dsl need to have a real compiler/executor, not just a validator. because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
I just have them write the tools to write the DSL's to do the thing then (most of) the sloppy code stays in the generator and if all the different things depend on each other they don't go stale and whatnot. And let them design the DSL themselves for whatever task so it matches their 'internal concept' of how the things work.
Worked out pretty well so far but not really practical unless your goal is to make the tools to make the DSLs to make jitting VMs -- https://github.com/dan-eicher/BBQ kind of snowballed from "let's parse some binary files" to a way over the top toolkit for playing around with this stuff but, it's fun...
> you are basically looking at a whole system prompt just describing the new language
whats wrong with this? You may be over-indexing on the need for large quantities of examples. These days self-play through RL is far more effective and data (not compute) efficient.
The situation is much better than you think. I actually have a little DSL project along these lines: it's meant to be a declarative component in a larger application which enforces certain data guarantees.
It is not based on JSON or any other out-of-the-box format. The only source material for this DSL anywhere in the world is on my laptop: a little documentation, a few examples, a partial implementation.
This is enough. Codex can not only give me arbitrary examples in this novel DSL on demand, it can see what I'm trying to do with the compiler and extend it for me.
What is the general consensus on Martin Fowler - I worked with Thought Works and they were obsessed with overcomplicating everything, but maybe that is just agency in general?
I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.
Ironically Martin fowler was among the wave of influencers that were trying to get away from complexity of enterprise software. remember EJBs lol.
That wave included
1. TDD by beck
2. spring framework
3. rails ( later )
4. Agile manifesto
5. refactoring by fowler
6. gang of four design patterns
That said. Thoughtworks is a moneygrab that tried to cash in on fowler brand. I worked with them at sears ( worked closely with author of article) and siemens. they were no different from any other consulting firms that try to overcomplicate things so they can deploy more warm bodies to the project.
I'm really starting to tire of people making broad, general claims about how LLMs work or how to use them with N = 1 or 2.
An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, maybe you could extrapolate hard from small samples, but they don't work or understand things like we do. You need to show that the behavior you are documenting is an average behavior the LLM converges toward in the long run.
i'm not sure it's clearly established LLMs can't be a model of some part of “the way humans do”?
to be more specific, i'd argue LLMs “understand” awfully similarly to a brilliant (polymath) with early dementia or Alzheimer's
no executive function, no short term memory, and absent both of those, conversing with that person about the past or with an LLM about topics that had been "in their training sets before a cutoff date" is surprisingly similar, right down to: introing a topic precisely the same way, you'll experience the same conversation; convo loops if bits are too quantized (looping and lossiness / recall / context-length are correlated); and ofc opening a new session is like the first one never happened
> to be more specific, i'd argue LLMs “understand” awfully similarly to a brilliant (polymath) with early dementia or Alzheimer'
At which point i'd argue that you're kind of refuting your own claim as most humans are not brilliant polymaths with early onset alzheimer's.
Even admitting these kinds of comparisons to edge case human mental experience, there are more differences than similarities, and the similarities are misleading and superficial. There are still a lot of differences, even when it comes to the physical structure of the brains neurons as compared to digital neural nets, and i think it's far more beneficial to try to understand these machine in their uniqueness and for what they are than to draw hasty and shallow comparisons. You could be forgiven for that, though, because computing is rife with people who love to draw hasty unjustified analogies for some reason (example: people were already likening the brain to a computer when we didn't even have working implementations of neural nets yet and a computer was literally just a small number of logic gates lol)
I frequently blur the line between ad-hoc DSL and pseudocode, and just hand it off to the LLM. I want to get the thoughts out of my head as fast as possible, using whatever structure makes sense to me. Even if you know all of the code to be written, I think this is a huge win with LLMs, where your intent is more important than syntax.
The only thing which enables reliable use of LLMs are statistical techniques.
Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically.
The fact that the major model providers keep publishing nano/mini/luna variants should be a massive hint that there's more to this than one big fat loop magically one-shotting everything.
This reminds me of this Bjarne Stroustrup's Rule (creator of C++):
- For new features, people insist on loud, explicit syntax.
- For established features, people want terse notation
Hillel Wayne [1] argues that the same applies for the differences between what beginners and experts desire from a language:
Beginners need explicit syntax, experts want terse syntax.
In my mind, DSLs are related to that – a short notation to avoid repetition. And LLMs are the experts.
I wonder if Lisp with its powerful DSL-creating macros will enjoy more popularity in the near future.
Rather than DSLs I’ve found careful force tools results force the same kind of discipline in a more straightforward way to implement. So it’s normal to ask the llm to answer only yes or no and they are pretty good at following that instruction but it doesn’t scale so well. Whereas if the shape of the force tool call gives them more richness without giving them freedom to go off piste it scales to more nuanced results whilst also being trivial to parse.
As much as I dislike them [1], DSLs are an easy way to provide domain specific backpressure to your llm which might not easily be available in your language.
What I do for my one framework (in dotnet, strong typing & compiler) is to create claude skills (bash scripts and dotnet console apps) that claude can call to do various things within the framework. I have a template engine (for classes/db scripts/view engine templates), build tools, deployment tools, backup tools, infra docs - all of it works amazingly well. I built them in a standardized way so it's easy to chain them together and claude just 'gets it'. It feels like a DSL on steroids.
I discovered this a few months ago when I was describing a binary data format using Dogma. On a whim, I tried handing the task off to Claude and it not only finished it correctly, but also fixed some errors!
Logically this makes sense, but in practice it doesn’t, at least in my experience with SwiftUI. The LLM isn’t any better about generating/understanding it.
While the DSL is more formal than natural language, it’s not what we’re communicating to the LLM with, so it’s advantages are washed away. And typical code is more strict/rigorous than DSLs so I think that’s why I see worse results, because a typical languages compiler “catches” more mistakes, versus a DSL that’s easy to write but has lots of implicitness.
I’ve had the same journey experimenting with levels if abstraction too. Going lower, and exposing the LLM to the “full-stack” works much better than trying to build up abstractions it can’t see into without extra steps.
I don’t want to be too much of a hater, but these types of panacea/architecture posts are usually written by people who don’t work in the field, lack pressure or constraints, and get paid to goof around in castles of the mind. I would simply skip over it and hold my comments/opinions to myself, but they tend to have an outsized influence on software engineering practices.
That’s basically LISP 101. Before solving a problem, you build out the spare parts and the tooling. And then the software building feels like assembling lego blocks. Building a DSL or a good set of symbols (functions/classes/constants/enum/…) is the cornerstone of DDD.
The actual implementation details only matters at the coding stage. At the design stage, it’s better to define the glossary and its semantic.
42 comments
[ 5.0 ms ] story [ 57.6 ms ] threadFor example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.
You'd think it wouldn't because, well, no training data, but a short spec is usually enough.
dsl stays small is doing all the heavy lifting here
the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (specially if its not json/yaml or just function chain based). as the number of things that can mix and match increase you are basically looking at a whole system prompt just describing the new language.
this brings us to the second part. step 2: after dsl is 'planned' (note they use the java compiler), the dsl need to have a real compiler/executor, not just a validator. because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
Worked out pretty well so far but not really practical unless your goal is to make the tools to make the DSLs to make jitting VMs -- https://github.com/dan-eicher/BBQ kind of snowballed from "let's parse some binary files" to a way over the top toolkit for playing around with this stuff but, it's fun...
whats wrong with this? You may be over-indexing on the need for large quantities of examples. These days self-play through RL is far more effective and data (not compute) efficient.
It is not based on JSON or any other out-of-the-box format. The only source material for this DSL anywhere in the world is on my laptop: a little documentation, a few examples, a partial implementation.
This is enough. Codex can not only give me arbitrary examples in this novel DSL on demand, it can see what I'm trying to do with the compiler and extend it for me.
Also this article is a good pre-cursor to the DSL article:
https://martinfowler.com/articles/what-is-code.html
I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.
That wave included
1. TDD by beck 2. spring framework 3. rails ( later ) 4. Agile manifesto 5. refactoring by fowler 6. gang of four design patterns
That said. Thoughtworks is a moneygrab that tried to cash in on fowler brand. I worked with them at sears ( worked closely with author of article) and siemens. they were no different from any other consulting firms that try to overcomplicate things so they can deploy more warm bodies to the project.
An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, maybe you could extrapolate hard from small samples, but they don't work or understand things like we do. You need to show that the behavior you are documenting is an average behavior the LLM converges toward in the long run.
> understanding anything in the way humans do
i'm not sure it's clearly established LLMs can't be a model of some part of “the way humans do”?
to be more specific, i'd argue LLMs “understand” awfully similarly to a brilliant (polymath) with early dementia or Alzheimer's
no executive function, no short term memory, and absent both of those, conversing with that person about the past or with an LLM about topics that had been "in their training sets before a cutoff date" is surprisingly similar, right down to: introing a topic precisely the same way, you'll experience the same conversation; convo loops if bits are too quantized (looping and lossiness / recall / context-length are correlated); and ofc opening a new session is like the first one never happened
At which point i'd argue that you're kind of refuting your own claim as most humans are not brilliant polymaths with early onset alzheimer's.
Even admitting these kinds of comparisons to edge case human mental experience, there are more differences than similarities, and the similarities are misleading and superficial. There are still a lot of differences, even when it comes to the physical structure of the brains neurons as compared to digital neural nets, and i think it's far more beneficial to try to understand these machine in their uniqueness and for what they are than to draw hasty and shallow comparisons. You could be forgiven for that, though, because computing is rife with people who love to draw hasty unjustified analogies for some reason (example: people were already likening the brain to a computer when we didn't even have working implementations of neural nets yet and a computer was literally just a small number of logic gates lol)
Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically.
The fact that the major model providers keep publishing nano/mini/luna variants should be a massive hint that there's more to this than one big fat loop magically one-shotting everything.
Hillel Wayne [1] argues that the same applies for the differences between what beginners and experts desire from a language: Beginners need explicit syntax, experts want terse syntax.
In my mind, DSLs are related to that – a short notation to avoid repetition. And LLMs are the experts.
I wonder if Lisp with its powerful DSL-creating macros will enjoy more popularity in the near future.
[1] https://buttondown.com/hillelwayne/archive/stroustrups-rule/
1- https://mikehadlow.blogspot.com/2012/05/configuration-comple...
edit: this is all on linux + posgresql.
While the DSL is more formal than natural language, it’s not what we’re communicating to the LLM with, so it’s advantages are washed away. And typical code is more strict/rigorous than DSLs so I think that’s why I see worse results, because a typical languages compiler “catches” more mistakes, versus a DSL that’s easy to write but has lots of implicitness.
I’ve had the same journey experimenting with levels if abstraction too. Going lower, and exposing the LLM to the “full-stack” works much better than trying to build up abstractions it can’t see into without extra steps.
I don’t want to be too much of a hater, but these types of panacea/architecture posts are usually written by people who don’t work in the field, lack pressure or constraints, and get paid to goof around in castles of the mind. I would simply skip over it and hold my comments/opinions to myself, but they tend to have an outsized influence on software engineering practices.
The actual implementation details only matters at the coding stage. At the design stage, it’s better to define the glossary and its semantic.