I have only read the first end of the article but I can't help but think that a project like libriscv[0] would've/could've worked for their game project too because fun fact but the creator of librsicv, the legendary fwsgonzo is also making a game. I highly recommend for people to check out their discord server.
But my main point is that libriscv is one of the fastest libriscv emulators and then something like C/C++/lua could've been used with sandboxing purposes for the purposes of the game then.
Am I missing something? Although, making a programming language is one kind of its own projects and that's really cool as well :-D
but I would also love to hear the author's opinion on libriscv as it feels like it ticks of all the boxes from my understanding
this project is pretty interesting, although i'm wondering how they're planning to address the "easy sandboxing" design goal in a compiled language with raw pointer arithmetic and clib interop... in that regard i think lua would have been a lot easier to sandbox, despite the author's concerns.
(also, they might want to look into lua userdata, since that would address their concern about the overhead of converting between native and lua data structures. the language is designed to be embedded in C programs after all)
Making you own language is easy. Creating the library that will actually solve problems without forcing the developers to reinvent the wheel is the crux. There is a reason why C++ / Java / JavaScript etc are established, it's the already proven libraries around those languages that allows them to be so successful.
I had a similar surprise about how approachable PL is, but from going from 'the bottom up' instead from a normal language.
I wrote a compiler toolchain and debugger that takes a Turing machine description plus input string and emits an encoded tape runnable by a Universal Turing Machine [0]. I had some prior PL experience, but never did an end-to-end compiler pipeline, at least not this low level.
It started as a joke/experiment, but I couldn't believe how fast it pulled me into designing:
- a small low-level ASM for building the UTM
- an ABI for symbol widths and encoding grammar
- an interpreter used as the behavioral oracle
- raw TM transitions for each ASM instruction, generated by having an LLM iterate on candidate emissions and checked against the interpreter oracle
- a CFG-style IR to fix the LLM mess once direct ASM -> TM emission became too hard to keep sane (LLM did a decent job actually, I don't think I would have done a much better job without the IR either)
- a gdb-style debugger for raw transitions, ASM routines, and blocks
- a trace visualizer
- a bootstrapping experiment where an L1 UTM/input pair was itself run through an L2 UTM
- optimisation experiments
And every step came quite naturally and was easy to tie in with everything else. Each one was just the next local repair needed to make the previous layer tractable.
I've been having a lot of fun building my own programming language [1]. Getting to the point where you can write programs in your own language was surprisingly easy.
The language, Sapphire, is Ruby inspired, so the most interesting part is digging into the internals of the latter when I'm trying to figure out how something should work.
I'm disappointed how many people just try to go for one more tiny incremental improvement or just make a programming language that suits their "taste", where "taste" just means similar to the programming languages they're familiar with (6 months+).
I just don't understand how someone can spend hundreds of hours building a programming language that is basically the same as any other, when they could have used a popular language for serious work and then try out a novel concept for their hobby language.
There is BerryScript which could have been a subset of an existing programming language plus a runtime optimized for that subset. There is also toitlang. They add hot reloading for microprocessors, which is good but that isn't enough to justify an entirely new programming language.
I wrote my own interpreted language about 25+ years ago to write online surveys. It made it easy to create complex surveys with many branches. I think I wrote it in Objective-C.
The team implementing the survey system wound up using the same language to implement the runtime portion, something I never expected or designed in.
I don't recall anything about what it looked like now. I do remember it was a lot of fun to write.
Yes, it's true that someone can put together a simple language like in a university course. The difficulties, as mentioned at the bottom of the post, are things like metaprogramming features or optimizing compilers.
The tail ends of a language implementation (parsing and code generation) are a fixed cost; the "middle end" can grow unbounded as more production-quality items are added.
Anyone trying to do this... the first thing you do is avoid lex/yacc/bison/antlr. You do not need all this ceremony. A recursive descent parser that uses Pratt parsing will work for a vast majority of cases.
The lexer/parser is never the bottleneck. In fact, you can write those two by hand over a single weekend for a largish language. With LLMs, it takes 15 minutes if you have an unambiguous spec.
The biggest time sink, and the reason you will fail for sure, is the inability to restrict the scope of the project. You start with a limited feature set and produce the entire compiler/vm toolchain. Then you get greedy and fiddle with the type system, adding features that you have never used and probably never will. And now you have to change every single phase from start to end.
I learned to do this about 2 years ago (pre LLM). I have been developing software for ~30 years and somehow doing something like this was a major mental obstacle, mostly created by the perception of "the dragon book", as in this topic being full of mystical unobtainable incantations, so I never even dared venture into this space. Silly, I know. However, after diving into this and learning to write a recursive descent parser for a DSL I wanted to write, it felt like I'd acquired a superpower. Totally understand that there is many more layers to all of this, layer that can get very complex, but just learning that first bit...
I agree. I have written lexer/parser for my language twice (for compiler0 and for a self-hosted compiler). It's a very dumb task requiring almost to mental load.
Profiling results show that the amount of time spent lexing/parsing is negligible - less than 1% of the total compilation time.
Probably the most fun I’ve had with LLMs has been slowly making a programming language as a side project.
I used to give up somewhere around the type system, too, but this time I’m approaching something vaguely useful. It even has a basic LSP.
It’s been both enjoyable and enlightening, and LLMs turn out to be an excellent pair designer as (in addition to implementation) they’re really good at summarising the impact of various decisions.
> the reason you will fail for sure, is the inability to restrict the scope of the project
This will be the reason, for sure. But then the scope of every project like this tends towards building an OS with it then replacing every piece of software, including all embedded devices :)
Many projects wish they had a proper grammar. When a project turns useful and people want to port it, or support it on other platforms, a grammar makes that job much easier.
I am not quite sure what you mean by having a recursive descent parser, because you can write one manually, or you can generate one from a grammar, which would have the additional benefits of having a grammar. I recommend having a grammar.
Yep. I started out using ANTLR for one project of mine. I ended up spending loads of time fighting its syntax to do really quite simple things, and it was slow! I probably wasn't holding it right. In the end, I wrote a simple lexer and recursive descent parser (with a small amount of lookahead) in a weekend. The code was easy to read, easy to extend, and fast.
The part where most people give up is useful error messages and parsing recovery. Parsing a correct document with a grammar is really easy, to the point that nobody should be under the illusion that it is difficult. The moment you add error recovery, the things that are obvious have n ways to go wrong and you have to come up with test cases for each of them.
To add, another reason to hand-write your parser is that it gives you much, much better opportunities for adding helpful diagnostics.
Grammar rules are not meaningful to your users - they don’t have the same mental model as whatever parser generator you’re using. If you want to be helpful to them, you probably want some pattern matching in the error path as well.
Best in class here is probably rustc, which has incredible UX compared to most compilers.
I feel the need to reference Crafting Interpreters design note on parsing here.[1]
To quote an excerpt: "If you’re just trying to get your parser done, pick one of the bog-standard techniques, use it, and move on. Recursive descent, Pratt parsing, and the popular parser generators like ANTLR or Bison are all fine."
In my opinion, parser generators are great if you just want to write a grammar and be done. Especially if that's what you learned in college and it's all you know.
Like most things in programming, handling the easy stuff is easy, but it’s all the edge cases that kill you. I’m writing an IDE in flutter right now, and all of the defensive programming I have to do to handle the unhappy path, is where 50% of my code goes.
I was very deep into .NET until recently but somehow I didn't know this existed. Looks like C# with extra Linq-to-SQL syntax; I guess it's a DSL made with Roslyn for ERP jobs? I wonder how they picked the name.
The compilers you get for both c++ and c are the best of any compiler. Clang is better than anything I have used in terms of the amount of options and features it has and also how polished it is
Just making a better C with no real compiler (only JIT) is easy, I agree.
It's much harder to make something innovative and mature. It requires years of development.
To me the most interesting part of a notation is the underlying thing that actually runs the code. The virtual machine, if you will. There are many ways to do that but I don't know a good systematic overview. E. g. what is Forth, if we ignore the notation? What is Lisp? What is Pascal and how it is different from C?
I've also made my own language for making games. It's a scheme with some tricks to make some gamdev specific aspects much nicer. Making it work was indeed not that hard, but making it good has taken its toll. Really happy with it currently!
I think more people having a crack at a language is a good thing. It demystifies a lot. For a long while I wanted the install guide for EYG (my language) to be a tutorial to write an interpreter in the language of your choice. I thought following the guide should take about a weekend and cover every feature in the language. For production you might want someone else's implementation, but for getting started what a great intro.
45 comments
[ 2.9 ms ] story [ 66.1 ms ] threadRoughly 100%.
But my main point is that libriscv is one of the fastest libriscv emulators and then something like C/C++/lua could've been used with sandboxing purposes for the purposes of the game then.
Am I missing something? Although, making a programming language is one kind of its own projects and that's really cool as well :-D
but I would also love to hear the author's opinion on libriscv as it feels like it ticks of all the boxes from my understanding
[0]: https://github.com/libriscv/libriscv
(also, they might want to look into lua userdata, since that would address their concern about the overhead of converting between native and lua data structures. the language is designed to be embedded in C programs after all)
I wrote a compiler toolchain and debugger that takes a Turing machine description plus input string and emits an encoded tape runnable by a Universal Turing Machine [0]. I had some prior PL experience, but never did an end-to-end compiler pipeline, at least not this low level.
It started as a joke/experiment, but I couldn't believe how fast it pulled me into designing:
- a small low-level ASM for building the UTM
- an ABI for symbol widths and encoding grammar
- an interpreter used as the behavioral oracle
- raw TM transitions for each ASM instruction, generated by having an LLM iterate on candidate emissions and checked against the interpreter oracle
- a CFG-style IR to fix the LLM mess once direct ASM -> TM emission became too hard to keep sane (LLM did a decent job actually, I don't think I would have done a much better job without the IR either)
- a gdb-style debugger for raw transitions, ASM routines, and blocks
- a trace visualizer
- a bootstrapping experiment where an L1 UTM/input pair was itself run through an L2 UTM
- optimisation experiments
And every step came quite naturally and was easy to tie in with everything else. Each one was just the next local repair needed to make the previous layer tractable.
[0] Repo: https://github.com/ouatu-ro/mtm
Seems like there might be some overlap with Tom Stuart's excellent Understanding Computation book - https://computationbook.com/
The language, Sapphire, is Ruby inspired, so the most interesting part is digging into the internals of the latter when I'm trying to figure out how something should work.
[1] https://github.com/sapphire-project/sapphire
Coming up with new ideas is hard. Especially since you have to test them in the real world.
I just don't understand how someone can spend hundreds of hours building a programming language that is basically the same as any other, when they could have used a popular language for serious work and then try out a novel concept for their hobby language.
There is BerryScript which could have been a subset of an existing programming language plus a runtime optimized for that subset. There is also toitlang. They add hot reloading for microprocessors, which is good but that isn't enough to justify an entirely new programming language.
The team implementing the survey system wound up using the same language to implement the runtime portion, something I never expected or designed in.
I don't recall anything about what it looked like now. I do remember it was a lot of fun to write.
I'm kind of curious and want to try it for fun as long as i get some free time ^^
The tail ends of a language implementation (parsing and code generation) are a fixed cost; the "middle end" can grow unbounded as more production-quality items are added.
My language: https://www.empirical-soft.com
The lexer/parser is never the bottleneck. In fact, you can write those two by hand over a single weekend for a largish language. With LLMs, it takes 15 minutes if you have an unambiguous spec.
The biggest time sink, and the reason you will fail for sure, is the inability to restrict the scope of the project. You start with a limited feature set and produce the entire compiler/vm toolchain. Then you get greedy and fiddle with the type system, adding features that you have never used and probably never will. And now you have to change every single phase from start to end.
I mostly give up at this stage.
Profiling results show that the amount of time spent lexing/parsing is negligible - less than 1% of the total compilation time.
I used to give up somewhere around the type system, too, but this time I’m approaching something vaguely useful. It even has a basic LSP.
It’s been both enjoyable and enlightening, and LLMs turn out to be an excellent pair designer as (in addition to implementation) they’re really good at summarising the impact of various decisions.
> the reason you will fail for sure, is the inability to restrict the scope of the project
This will be the reason, for sure. But then the scope of every project like this tends towards building an OS with it then replacing every piece of software, including all embedded devices :)
I am not quite sure what you mean by having a recursive descent parser, because you can write one manually, or you can generate one from a grammar, which would have the additional benefits of having a grammar. I recommend having a grammar.
Grammar rules are not meaningful to your users - they don’t have the same mental model as whatever parser generator you’re using. If you want to be helpful to them, you probably want some pattern matching in the error path as well.
Best in class here is probably rustc, which has incredible UX compared to most compilers.
To quote an excerpt: "If you’re just trying to get your parser done, pick one of the bog-standard techniques, use it, and move on. Recursive descent, Pratt parsing, and the popular parser generators like ANTLR or Bison are all fine."
In my opinion, parser generators are great if you just want to write a grammar and be done. Especially if that's what you learned in college and it's all you know.
[1]: https://craftinginterpreters.com/compiling-expressions.html#...
Only thing that goes for C++ is that it has acceptable (not straightforward) C interop.
I don't like C# and X++ because the language surface is huge but if you use a limited subset than needles to say, very useful and handy languages too.
I was very deep into .NET until recently but somehow I didn't know this existed. Looks like C# with extra Linq-to-SQL syntax; I guess it's a DSL made with Roslyn for ERP jobs? I wonder how they picked the name.
Nah, that's far from the only thing.
Maybe AI is good enough now to help me with that..
The last time I tried, Claude couldn't even help me build a syntax highlighter for a hypothetical language.