I have also been working on a programming language, but longer than the author of this piece (since 2011), and the author is correct.
Designing a language consumes you; you never see programming the same way again. Whenever I write C (I'm implementing the interpreter in C), I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language.
And the design never stops. As mentioned, I've been designing mine since 2011, and I'm only now writing an interpreter after two failed attempts to bootstrap a compiler.
So the author is correct: don't write a programming language unless you are prepared to be marked and scarred for life.
I was looking for your comment in this thread and I'm not disappointed.
It's fortunate you went ahead despite every thing telling you to do otherwise. And then tricked other people into joining you. A bit evil probably but they consented, in the end. You are probably only partly responsible.
It's funny/interesting that an 1kloc implementation of minimal self hosting Go exist, or C/pascal/oberon compilers ; perfect hindsight view etc. You theoretically could (well if nobody does, no you can't) create a nice language and its implementation quickly.
> It's funny/interesting that an 1kloc implementation of minimal self hosting Go exist
Does it? That's not the technique Go introduced, they started by rebuilding Go from earlier versions written in C - but in fact the bootstrapping process may change soon, see https://github.com/golang/go/issues/44505
Sure, go itself didnt build it from scratch - rewriting your implementation each time you change your language might grow tiresome fast, but I was referring that toy/minimal implementations exist : https://benhoyt.com/writings/mugo/
Wirth was even more drastic in what concerns writing bootstraped compilers.
I don't recall in what paper from him I have read this, so take the story with a grain of salt, also open to corrections.
He actually wrote the initial version of the compiler directly in Pascal.
How did he perform such thing, one would ask.
By writing it on paper using a Pascal subset good enough as stage 0, and then manually compiling the source code into corresponding Assembly instructions that he would then actually type into the cards.
So when he finally got the compiler done, it was already bootstrapped from the get go.
Additionally, P-Code originally wasn't designed to be an interpreter, rather to repeat the above process in an easier way across computer systems.
He was initially surprised that others took it to write Pascal interpreters instead of a bootstrapping tool.
I've never been megalomaniac enough to think I could design a new language. But I did once write a Pascal source-level debugger; I think the compiler was Intel Pascal (long time ago). It taught me a lot about how the compiler worked, and how the language worked.
I've used to suffer quite badly from Impostor Syndrome, and that project did a lot to alleviate my symptoms.
I never shared the project with anyone else; it was never completely finished.
My hard drive is full of unfinished projects, started to learn about programming language, or try out new design approaches, quickly abandoned after the initial goal was achieved.
Regarding imposter syndrome, two guidelines that have helped me in such situations, are "only repent for paths not taken", "failure is better that not knowing at all".
> I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language
That is exactly what motivated me to start thinking about my own language and what made me want to keep working on it in the early to mid 1990s. Luckily, I started learning several languages so that I'd have a better idea of the design space. Someone suggested I take a look at a relatively unknown language called Python because it fit how I was thinking about things. He was right and that was the anecdote for me - finding a language that fit how I thought about programming well enough to eliminate enough of the annoyances that were my motivation.
The lesson... If you're thinking about designing a language, study as many other languages as you can so that you get a better sense for the design choices involved and how various concepts fit together. Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly. If you get lucky, you might find a language that stops your descent into the rabbit hole.
The above might only work if you can accept that languages are sufficiently complex that they all have real problems, even yours when it's "done".
> If you're thinking about designing a language, study as many other languages as you can so that you get a better sense for the design choices involved and how various concepts fit together.
I've written C, C++, Python, Ruby, Lua, Bash, POSIX sh, C#, Java, JavaScript, Common Lisp, Racket, Haskell, bc, dc, Zig, BASIC, Tcl, and some others I'm probably forgetting, and I've studied Rust, D, Ada, Idris, Coq, Isabelle/HOL, HAL/S, OpenGL shaders, Cilk, Cyclone, Ceylon, Scala, Clojure, Kotlin, F#, Scheme Lisp, Jai, Julia, Perl, Nim, and probably even more that I am forgetting.
Does all of that count? :)
> Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly.
Yes, I agree. And I have done all of that.
> If you get lucky, you might find a language that stops your descent into the rabbit hole.
"Lucky" is the operative word. Unfortunately, I'm picky and haven't been that lucky.
It is interesting to see such an extensive list that doesn't include Smalltalk or Forth. The lack of Pascal/Modula-2/Oberon is not a problem given Ada. Not having APL (or J or K) might not hurt if you get the needed ideas from Julia. Other languages such as Cobol (or the microcomputer equivalent such as dBase) most people can probably live without.
Ah, whoops. Smalltalk and Forth are two languages that I have written a bit of code in and studied extensively. I did say that I was probably missing some.
On the bright side, it probably better equips you to actually argue about where C (or any language) has faults from a language design perspective, which is a relatively rare point of view.
> So the author is correct: don't write a programming language unless you are prepared to be marked and scarred for life.
I write and wrote internal languages mostly; a bunch of them made it to production inside larger (for my country) companies for years and even a decade. The scarring part is that I always add the features I want and when I go to a mainstream language, I always miss those features. So since 2018 we have been doing a new language and one of them will be publicly launched this or next year so I can at least use that for day to day programming without being annoyed by missing features. Note that I already have that using Racket with a bunch of macros but in my business it is easier to introduce your own language than use Racket...
Most people are allergic for lisp likes. Not sure why but it has since always been blocked. I even did a lispy but with <> instead 20 years ago and that was OK (it was the xml for everything hype) while with () it was rejected. I had a talk with the cto of one of our bigger infra structure partners and he had exactly the same issue, so he replaced the () with [] and suddenly it was fine. So he has had scheme in their products for over 10 years but with [] because people are allergic to ().
I can obviously do the same with racket but if no () then I rather have another syntax all together.
Ah, it's sad to hear that. People can be so emotional, just because of a bit of syntax.
But since you have full control, you could introduce some extra syntactic sugar as well and desugar it into regular lisp/racket syntax before parsing, no?
As an alternative with a completely different syntax, I found Scala to be extremely nice for building DSLs. Probably the best typesafe language for DSLs out there. It's only really good on the JVM though.
> I've been designing mine since 2011, and I'm only now writing an interpreter after two failed attempts to bootstrap a compiler.
You are lucky (or unlucky) that you have reached that stage. I'm trying since 2004 and had at least 6 failed attempts on design and 3 different implementations that never finished. Fortunately for me, since I had a lot of failed designs (as opposed to implementations) I better appreciate existing designs even with their warts.
Article author here. The two years mentioned in the post is the time spent actually implementing it. I spent a few more years thinking about it prior, and I thought that implementing the basic ideas would prove, our not, whether it worked, and then I'd be done.
Well, guess what, it didn't actually turn out that way. At least I'm more understanding of people who annoyingly promote their own languages all the time.
> I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language.
What's worse is that PL authors are usually students of PL history, and the history of PLs is littered with "We did things this way out of necessity. We wouldn't have done this if we had more time/resources." Which would be fine but then we are all stuck using those tools with all their arbitrary limitations and quirks for 50 years. If you ever thought "I could write something to make this so much easier", the impulse is 100x worse as a PL designer, because you realize how many things we do are done for no reason at all beyond history and happenstance.
I never really got into it that deep: I thought i should
enjoy it, but after writing a small toy language I realised I don’t. What I have realised is I get much more joy from
building things than building the tools but that is just me. Im having much more fun knocking up stuff in nextjs! I am glad there are people who love building the tools and frameworks though such that I am spoilt for choice.
The problem with writing a _general-purpose_ programming language (as a solo-developer), is the fact that writing the language (i.e defining the syntax+semantics, writing the parser/compiler) is only half the battle - the rest of your time is spent on less technically interesting QoL features, like an IDE, standard library, debugger, syntax highlighting etc. which are nonetheless vital for making the programming language actually usable. (I still think writing your own PL is a useful exercise just to see how the sausage is made, but it should be made clear that there is a world of difference between a toy language and one that people would actually want to use)
It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free, leaving you more time to work on designing the language to suit your needs. If you combine macros with clever uses of GADTs you can even get the host language to "type-check" programs written in your DSL. OCaml and Lisp/Racket are my favourite host languages to use for this approach.
Yeah it's pretty bad these days. If you tell someone you are writing a PL, they expect:
- a complete parser/compiler and debugger toolchain
- a complete language server implementation, and language modes for all popular IDEs including VSCode, VIM, Emacs, etc.
- a package manager with a large, secure, and vibrant package ecosystem
- a community with live chat support and a strong Stack Overflow presence
- robust documentation with copious examples, tutorials, guides, and technical info. These days even a multi-hundred page book with professional editing available for free is expected.
- That you will be responsive to bugs, provide patches promptly, and review/merge community feature requests asap. They will get very mad with you if you don't, and likely write a blog post about how much of a tyrant you are over your project.
- your language is expected to have undergone a full independent security audit, and to be up-to-date on the latest security issues. It's expected to be stable with complete backwards compatibility.
- you have to manage the community and deal with petty interpersonal conflicts or else word gets out that your language has a "toxic community"
- support for all major operating systems and hardware architectures including Mac, Windows, and all distributions of Linux, as well as esoteric hardware.
- support for the web through wasm, so you need experts at that as well as x86 and ARM platforms now.
- And on top of all that, your project needs to be completely open source, with a free and open license that permits royalty free, patent free, commercial use. Oh, and your users both corporate and personal expect your work to be completely free as in beer. They won't pay anything for it. Not even a dollar. And if you ask them to pay something they don't weigh the value of your product versus the competition, they balk immediately and don't even consider it. It's been a non-starter to ask for money for a PL for decades.
It used to be the case that you could write down a spec on paper and call it a PL. Back in the 60s you could call that a "programming language" without even writing an implementation of the parser. Today a programming language is a massive undertaking and the saddest part of it all is that it's been commoditized. I've even seen a sentiment out there that programming languages are done, that we've invented all we need to. All future languages are unnecessary, and all current languages are largely the same. Could you imagine? So as hard as programming languages are, DO write them. That's my message. Just do so responsibly and with clear eyes and expectations.
At very least, because the baton eventually needs to be passed on to younger generations.
Regarding language innovation, I will oversimplify and assert that due the resistance to modernization (for whatever reasons), we have spent the last decades making Algol 68, Lisp and ML appealing to mainstream, smuggling their capabilities into more acceptable packaging.
I do however feel that your QoL will depend on whether or not you make a compiler or a translator. The key benefit of a translator is that you can co-opt the host language's standard runtime and debugger.
Agree with all of this. Don't do any yak-shaving you don't want to do just for the fun of it; build your language and its tooling on existing systems so you can spend your time on what makes it special, instead of what doesn't.
> It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free
I don't understand what you mean by this, so I'm hoping you can clarify. What's the advantage of doing this over the alternative way: writing a parser which would compile to code in the host language, or some primitives that you've arranged in the host language that would represent the bytecode of your language interpreter. From there you can leverage whatever in the host language you want, including its type system, packages, language semantics, FFI access to other languages. Really whatever you like.
What you're suggesting is leveraging a macro system to help you write what is essentially a parser/compiler, which is a good win. But I don't see what that buys you past that point. If I do things the other way, by writing a parser that consumes a *.my_lang file and compiles it to primitives or actual source code in the host language, I can still leverage all the benefits of the host language just as you describe doing with a macro system. The advantage of doing things this way is you're not limited to languages with tolerable metaprogramming capabilities.
I've never written a programming language. I have built a number of constructed languages[1] over the past few decades, which is an equally selfish (and thankless) task.
Less tangentially, I've worked for the past 8 years on building a HTML5 canvas library in Javascript[2]. I don't ever see people talking about JS libraries in terms of computer language design but I think, in a way, that is effectively what a (good) library is doing: building new, sometimes more developer-friendly, ways of conceptualising and creating programs that do stuff using novel APIs, syntax, state, etc.
There's a lot in the OP that resonates with my experience:
- "once you design a programming language you fall down a rabbit hole" - I've lost 8 years of my life to my project; but I've also gained 8 years of learning skills and insights that do translate to my everyday work as a frontend developer. For instance, almost all of my knowledge about accessibility comes from trying out many (many) different approaches to making the canvas element more accessible.
- "You keep adding new features that are really cool" - Guilty! My library doesn't need an integrated reaction-diffusion engine, but somehow it's acquired one along the way. Similar story with the reduced palette filter which, while a lot of fun to build and play with, remains too slow for use in production eg dithering live-stream video.
- "It also affects your communication with the rest of the programming community" - Because of the need for speedy calculations, I've developed an approach to programming Javascript which tends to avoid slower functionality. This sometimes leads to me writing code that could/should be written in a clearer, more concise and understandable way: 99% of JS code doesn't need to be super-fast, just fast enough!
I did design a language and it is still being used. I did it as part of my job at that time. It started as a simple query language inside the application we were developing. One by one features were added. Some colleague also decided to used it program certain functionality of the application, which caused more and more people to use it.
I implemented it with a parser that I developed myself. So, actually, I fell in the trap of developing a language myself, not a programming language, but a developing my own language developing tools. And I have doing it for more than 20 years now, on and off. I do recognize the self promoting bit. (I am kind of doing it right now.)
I think that a big tenant of any modern language should be interoperability, such that you can easily import/export code from another popular language. That not only clears up the issue around libraries, but also makes it a safer bet for usage by people who otherwise wouldn’t be able to use a niche language
The language I am refering to, is now called the BiZZdesign scripting language. See: https://support.bizzdesign.com/display/knowledge/BiZZdesign+...
The pipe operators are from the original design of the query language. Variables, function, and many other language constructs have been added later.
I can feel the author pain, even though I didn't invent a programming language, I just wrote the interpreter for one (specifically a Lua interpreter written in C# - moonsharp).
Why I did it: in 2013-2014 I was SUPER bored and burned-out, I needed a challenge and I picked that one. Oh, I was wrong.
The GOOD: the challenge has been fun, I learned A LOT. I learned how to use ANTLR, and then I learned that ANTLR was getting in the way and now I'm wary of ever using a parser generator at all, for sure not that one. I learned how to write a recursive descent parser, even if I did it by reversing engineering (i.e. mostly copying) the original rd-parser of puc Lua. And I learned that no matter what I do, what I find the hardest is respecting the correct operator priority in math expressions. I know that shouldn't be the hardest thing, but brains are fun.
It also ended up getting used. It was used in the OnePassword client for windows, and it's for sure used in the Shenzen IO and TIS-100 videogames. I'm sure there are many more users using that. It's stuck somewhat in between of "nobody uses it" and "it has enough traction that the community will take it forward and you can forget it even existed", and that, turns out, is not a great place to be.
The BAD: I picked C#, as that's the language I was using at the time and I had an interest in seeing it used in Unity. Big mistake. Don't get me wrong: C# is a great language and if you're planning to base your company on it, I don't see big reasons not to (I wouldn't, probably, but every language has its pros and cons). But C#, and in 2014 to boot, was a mistake for such a project. First, however multiplatform C# became, a lot of development is still Windows only (at least, if you want to target the main frameworks). And second, 2014 is precisely the year when C# started changing radically with the core / standard frameworks. The situation now is probably more stable, but in 2014/2015 it was a mess, and half of the reflection frameworks were broken to say the least.
The UGLY: I just didn't consider how much punches life can hit your face with. I changed job, twice. Changed platform from Windows to MacOS, to Linux, to MacOS again, now I program for multiple platforms in Rust most of the time, and my interest in C# is close to non-existent. Family members passed away, new kids were born, situations changed, and life hit me hard enough. Energy got focused on daily jobs, or on other side projects. Every other side project, by the way, has been met with humongous amounts of guilt, because I'm/was leaving moonsharp users (however few they might be) behind.
So where I'm now? I tried to delegate the project to other people, but I'm not effective at it (maybe they are, but in reality, I'm just absent).
I tried to get back active into the project but failed a couple of times and now I fear doing it again and creating expectations.
Sometimes I feel like contributing something, but I can't even get a proper dev setup running. I don't remember details well enough to not break something when fixing major bugs, nothing major but usually it goes over the amount of energy that I would like to spend.
Sometimes I read some question on MoonSharp, and I try to answer, but more often I read a question and before answering I have a guilt trip because I'm answering to this question and left alone all the others and so I simply avoid this one too.
Most of all, the project is not aligned anymore to my interests, so it feels like a chore.
So, if you are reading this: think twice before making the same mistake. If you want a side project, pick one that is either small, or has a clear end to it.
I find writing a library is usually enough for the types of problems I encounter. You can even design it to be language-like using fluent interfaces; take FluentAssertions for example: `actual.Should().StartWith(expected);`.
Beyond that, I tend to reach for:
- A config file. Can the problem be solved declaratively through a config file that a program reads and bases its behavior on?
- Templates, generators, and macros. Along the lines of syntax, would templates or generators solve problems with an existing GPPL?
- A domain-specific language. Does the problem lend itself to composable patterns or statements? Could a DSL help organize and simplify them?
- A transpiler. Is the syntax of an existing GPPL the problem? Would syntactic sugar be enough?
Of course, there are plenty of other reasons to write a new GPPL, like for your own learning or just as a hobby. Just know what you're getting yourself into.
Configuration files are such a useful thing to start with - I remember one of my most productive changes was to hack on a local copy of mutt:
* If the configuration file had mode 0755 then execute it, and parse the output.
* Otherwise read as-is.
That was my solution to writing a complex configuration file, which behaved differently upon different hosts - write the thing in perl, and have it "print" blocks and configuration values appropriately.
The same arguments could be applied to almost any field of research. If you get to spend your time doing something that gives you pleasure and challenges you, then it's worth that time. If it results in something that is also of use to others, all the better. There are countless examples in history of how people have worked on something, often despite lack of understanding or even rejection by others, and that very work later became very relevant, often many decades later. You could have spent the same amount of time watching TV instead of writing a programming language. Wouldn't you have found that an "utter waste of time" as well?
Who cares? I'm not writing my programming language for you: I'm writing it for myself. It's about the journey, not about the destination. (Btw, the author is right)
> The answer to the Infamous Question is always the same, and if the weasels ever ask it of you, snap back as quickly as possible, "Because I want to know how it works."
Literally why I took my first CS course. I just had to know how computers worked. Pure curiosity drove me.
Loved Jeff's book on x86 assembly back in the day. I still remember him (in the book) introducing a DOS interrupt that I could invoke I think through DEBUG that rebooted the computer. And in the text he says something like that: that's the feeling of power. And he was right, that was exactly how I felt.
The same goes for an OS or a game engine or any other large scope (Or collection of small scope) tool.
The other people who kinda care are those similarly engrossed in their own project, meaning... they do not actually have time to care.
My first priority with computing is to get rid of all the custom stuff. I won't even think about refactoring or any other improvement till I've weeded out all the handwritten reimplementations of library functions, made sure that there's no ad hoc string.split() based data formats that should be JSON, or UDP packets that should be MQTT.
Honestly, I don't even like that macros exist. I understand that in low level languages sometimes one needs metaprogramming, but it's more of an unfortunate reality than a cool feature.
What does "sanity" mean? Without the curiosity to drive us, we wouldn't have moved out of chilly caves.
Yes, like most initiatives, you are set for failure, if you define success by how popular a creation of yours becomes. You can still learn a lot.
And still, because "success" is such a rare breed, we actually need more people to do challenging stuff, so we'd get more success in absolute numbers.
The OP actually lacks sanity in recognizing that their project did not take off, and they don't have a clear idea of why they are doing their programming language project.
Basically, they are realising it's not a success, and they don't necessarily care about learning much out of it, and yet are still not letting go. If anything, that's a definition of insanity :)
It's more like addiction than insanity. They aren't doing the same thing and expecting different results exactly... They're doing the same thing just... because it's part of their identity now, even though they aren't all that happy.
Real insanity would be still thinking it was gonna be the next big hit.
I think we'd get more successes if people didn't didn't do this stuff and instead joined existing projects, or started with the business stuff first and tried to start their own foundations and such.
There are big wins, but they're incredibly rare and seem very random. And many of them are almost guaranteed to fail.
You'll learn a lot, but what can you actually use that knowledge for? Niche DIY stuff doesn't teach you much about mainstream methods.
The result is 1000 garage tinkerers for every Guido Van Rossum, and sometimes it's even actively detrimental to a project to try to cater to the tinkerers.
> I think we'd get more successes if people didn't didn't do this stuff and instead joined existing projects, or started with the business stuff first and tried to start their own foundations and such.
I think that you're working from a very different definition of "success" than your parent comment. "Success" doesn't have to mean "contributed to GDP". For some people "success" means "I had fun, learned stuff, and expanded myself as a person".
I'm using the definition the original article seems to imply.
Not so much contributed to the GDP, but "Advanced the state of the art in some way" or "Solved a problem of interest outside the programming community".
I would imagine only the very most dedicated would actually enjoy doing a new language, maybe a third of programmers, and probably only those who fully accepted their thing was probably not going to be an enterprise grade solution.
The issue I have with your approach is twofold: the 1000 garage tinkerers are harmless, and your approach forgoes the Guido Van Rossums, Rich Hickeys, Chris Lattners and so on.
I've met a bunch of people who do things like write their own typed concatenative language. So I can tell you, first of all, that they tend to spend the week merrily contributing to, say, orchestrating Kube clusters, and second, the time they spend bit banging their pet programming language wasn't going to go towards more orchestrating Kube clusters. Or whatever it is you think is actually important.
That's fine, but it seems like the target audience of the article is the people who hope to become the next Van Rossum.
It might even be aimed at people deciding whether their team at work should pursue a new language for some specific task.
Maybe this could be solved with a special badge that tells people which of the Three Tribes of Programmers you're writing for...
For purely toy projects, I'm not sure if any code blog advice at all is that helpful, except possibly "Make stuff really small if you want to impress other programmers".
Best Practices don't apply the same way to this kind of thing, since in a real industry project the best practice would probably be "Just don't".
If you mean a Domain Specific Language, i think that's precisely what the author is advocating for. If you can make a good library or DSL, there is no reason to invent an entirely-new programming language: you'll spend less time reinventing the wheel and more time delivering on high-level features.
It is true: it's a deep pit you'll enter. But the conclusion is wrong. If you feel it's fun -- then do write your own programming language. Even if you won't finish (which is likely). But you'll think about things that are a lot of fun, and you will have more clarity about what you do not like about existing languages, and why. And you'll learn that there are things you need to solve that you did not think about before. That some things cannot be combined, and that you will have to choose.
It is a very good experience, there is a lot to learn. As long as your goal is not to get it done quickly...
The key to writing a PL and surviving is to have an exit strategy. Figure out ahead of time how long you will spend on this project, otherwise it’s liable to last half a decade or more before your interest wanes.
PL design is like crack for programmers. It is a lot of fun, which is why you see hundreds of little languages posted here every month.
The danger is that some people get don’t have a clear idea. About what it takes to write a pl. They can get quick wins with a parser generator and maybe some easy semantics, which give a good taste of how fun PLs are but also belie the fact that the hardest parts are exponentially harder than that.
Anyone embarking on a PL needs to be clear eyed about a few things.
1. You’ll never make money off of it. No one sells compilers anymore, and there are like 2 big for-profit languages out there out of thousands. If you do make money, let us all know how.
2. The odds are your language will have exactly one user, yourself. Get comfortable with the possibility you will spend years on something and no one will really appreciate it.
3. The real work in creating a language is not technical but community building. Languages need people who speak/write it to thrive, so as much or more work needs to be put into the community effort for the language to succeed. If all you want to do is work on technical problems that’s fine, but technical prowess doesn’t make a language popular.
4. Get ready for criticism. Every programmer has their opinions about languages and they will be sure to tell you their opinion of your language. It will not be good. The only people who will appreciate your language are other language designers. Most everyone else will start by telling you how you’re doing things the wrong way, and they’ll end by asking why you’re eating your time with all this anyway. Get ready for that and steel yourself against it.
Matlab and Mathematica. There are probably more, and Microsoft makes a ton of money with .Net of course, but I'm thinking more along the lines of pure PL products and not something in a vast enterprise software ecosystem like Microsoft offers.
I’ve been spending a lot of time myself designing and prototyping my “ideal” language (mainly to address gripes with other languages) and I totally agree with the OP about it being a near total waste of time. But at the same time, albeit being a gargantuan amount of work it helps you learn alot about programming up and down the stack. Good language design requires lots of thought, devotion, trial and error and self moderation to come down to something consistent and coherent that’s not a mess of all the cool little features you thought it would be a good idea to put in.
This argument makes no sense. Just because I can’t implement new plumbing in my house myself doesn’t mean I can’t complain about my neighbor doing a shit job with his. This board is a Tower of Babel with lots of pointless languages - just because you can doesn’t mean you should.
I'll take the bait and provide evidence on why you should write a programming language.
First, it's technically very difficult, but you will gain deeper insight into the art of the craft. So, if you are a TC chaser or career minded person, then spending half a year writing a language will help you master the coding aspect of the game. I've started many languages since I started college, and each one was instructional. (I'm now 40 and an early retiree)
Second, it's fun.
Third, it may turn into something new. If some people don't write a new programming language, then we are stuck with what we have. This advice basically admits that the status quo is good enough.
The authors saying that the language, as a project, is a lifetime appointment? Well, this reveals everything. I believe if you want to do a programming language, then you must be willing to invest at least a decade or two.
So, here, I am at forty preparing to launch a SaaS around a language that I designed ( http://www.adama-lang.org/ ). The kicker, I believe, is that a project like this requires wandering the desert alone for quite a while.
I'm preparing to launch, and I just started to load test my shiny new production cluster. Low and behold, it sucks. Fortunately, I have a tremendous number of dashboards and isolated it to how I'm interacting with RDS. I've got my work cut out for me which I'll write about.
However, I have a potentially interesting business precisely because I evolved a language which solved a niche use-case. The number of problems that I have had to solve up to this point is not for the faint of heart. Life and reality are harsh mistresses.
So, maybe, yes, you can save yourself some heartache by not writing a language. Perhaps, a better way to think or phase this is "Writing a programming language is a lonely affair that will most likely end in tragedy after a long death march".
With respect to your first point, I'm reminded of Peter Alvaro's comment "the only good reason to design a programming language is to shape your understanding of the problem". He points out that he was the only user of his first PL, but he had no regrets of about that, because it shaped all his later thought in his research.
> Third, it may turn into something new. If some people don't write a new programming language, then we are stuck with what we have. This advice basically admits that the status quo is good enough... The authors saying that the language, as a project, is a lifetime appointment? Well, this reveals everything. I believe if you want to do a programming language, then you must be willing to invest at least a decade or two.
Sounds like someone should persuade some big-name university to hand out honorary PhDs to people who build new languages that produce demonstrable value (even if the value is not business value).
To solve a problem about 23 years ago I wrote a programming language + interpreter. It was meant to be a simple language to write phone surveys in, so basically a kind of DSL for a single purpose. The programmers who built the survey taking system that would use my language discovered they could write much of it in the language itself, which surprised me a lot. Was a really fun project. No idea what happened to it.
Writing a programming language is easy, when compared to making an actually useful programming language, which again is easy, when compared to actually maintaining and developing a programming language.
My heuristic is that the difference in work required is around three orders of magnitude between creation, usefulness and maintenance.
That being said, I think every programmer would benefit greatly, just by attempting to write a language.
Most people will not need or want to design or implement their own language.
However, it is not as hard as the author suggests. First, there are excellent tutorials, textbooks (Wirth, Dragon book) and tools available (lex, yacc, LLVM, CUP, Jflex, Lemon, ...).
You should give it a try just for the fun of learning.
You will not often _need_ to do it, as most capabilities can perhaps be realized as libraries (including network and mobile functionality mentioned by the author).
Once you design something useful, other people will likely chime in and advertise for your, write documentation etc.
Most complex software system incorporate compiler techniques or Domain Specific Languages (DSLs), for parsing configuration files or as a basis for building higher-level functionality (AutoLISP -> AutoCAD, ELISP -> Emacs).
He shouts out to Nanopass, a way of writing highly compositional compilers developed in the Scheme world and much ported elsewhere. This kind of approach much reduces the pain of designing and maintaining compilers and can lead to very efficient compilers: it is the technology used in Chez Scheme. It's not got much take-up outside of the Scheme-aware world, despite the existence of many ports of the essential idea to other languages; I note that interesting Nanopass links keep getting submitted to HN without getting commented on: https://hn.algolia.com/?q=nanopass
133 comments
[ 5.0 ms ] story [ 216 ms ] threadDesigning a language consumes you; you never see programming the same way again. Whenever I write C (I'm implementing the interpreter in C), I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language.
And the design never stops. As mentioned, I've been designing mine since 2011, and I'm only now writing an interpreter after two failed attempts to bootstrap a compiler.
So the author is correct: don't write a programming language unless you are prepared to be marked and scarred for life.
I can vouch for that! Fortunately, I have a great team to collaborate with.
It's fortunate you went ahead despite every thing telling you to do otherwise. And then tricked other people into joining you. A bit evil probably but they consented, in the end. You are probably only partly responsible.
You "just" have to get to the point where your compiler is written in your own language. This may take a while. Go got there in V1.5:
https://go.dev/doc/go1.5
Does it? That's not the technique Go introduced, they started by rebuilding Go from earlier versions written in C - but in fact the bootstrapping process may change soon, see https://github.com/golang/go/issues/44505
I don't recall in what paper from him I have read this, so take the story with a grain of salt, also open to corrections.
He actually wrote the initial version of the compiler directly in Pascal.
How did he perform such thing, one would ask.
By writing it on paper using a Pascal subset good enough as stage 0, and then manually compiling the source code into corresponding Assembly instructions that he would then actually type into the cards.
So when he finally got the compiler done, it was already bootstrapped from the get go.
Additionally, P-Code originally wasn't designed to be an interpreter, rather to repeat the above process in an easier way across computer systems.
He was initially surprised that others took it to write Pascal interpreters instead of a bootstrapping tool.
I've used to suffer quite badly from Impostor Syndrome, and that project did a lot to alleviate my symptoms.
I never shared the project with anyone else; it was never completely finished.
My hard drive is full of unfinished projects, started to learn about programming language, or try out new design approaches, quickly abandoned after the initial goal was achieved.
Regarding imposter syndrome, two guidelines that have helped me in such situations, are "only repent for paths not taken", "failure is better that not knowing at all".
Obviously those quotes aren't from me.
That is exactly what motivated me to start thinking about my own language and what made me want to keep working on it in the early to mid 1990s. Luckily, I started learning several languages so that I'd have a better idea of the design space. Someone suggested I take a look at a relatively unknown language called Python because it fit how I was thinking about things. He was right and that was the anecdote for me - finding a language that fit how I thought about programming well enough to eliminate enough of the annoyances that were my motivation.
The lesson... If you're thinking about designing a language, study as many other languages as you can so that you get a better sense for the design choices involved and how various concepts fit together. Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly. If you get lucky, you might find a language that stops your descent into the rabbit hole.
The above might only work if you can accept that languages are sufficiently complex that they all have real problems, even yours when it's "done".
I've written C, C++, Python, Ruby, Lua, Bash, POSIX sh, C#, Java, JavaScript, Common Lisp, Racket, Haskell, bc, dc, Zig, BASIC, Tcl, and some others I'm probably forgetting, and I've studied Rust, D, Ada, Idris, Coq, Isabelle/HOL, HAL/S, OpenGL shaders, Cilk, Cyclone, Ceylon, Scala, Clojure, Kotlin, F#, Scheme Lisp, Jai, Julia, Perl, Nim, and probably even more that I am forgetting.
Does all of that count? :)
> Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly.
Yes, I agree. And I have done all of that.
> If you get lucky, you might find a language that stops your descent into the rabbit hole.
"Lucky" is the operative word. Unfortunately, I'm picky and haven't been that lucky.
Oh, and Go. I've written some Go.
I write and wrote internal languages mostly; a bunch of them made it to production inside larger (for my country) companies for years and even a decade. The scarring part is that I always add the features I want and when I go to a mainstream language, I always miss those features. So since 2018 we have been doing a new language and one of them will be publicly launched this or next year so I can at least use that for day to day programming without being annoyed by missing features. Note that I already have that using Racket with a bunch of macros but in my business it is easier to introduce your own language than use Racket...
Why is it easier?
I can obviously do the same with racket but if no () then I rather have another syntax all together.
But since you have full control, you could introduce some extra syntactic sugar as well and desugar it into regular lisp/racket syntax before parsing, no?
As an alternative with a completely different syntax, I found Scala to be extremely nice for building DSLs. Probably the best typesafe language for DSLs out there. It's only really good on the JVM though.
You are lucky (or unlucky) that you have reached that stage. I'm trying since 2004 and had at least 6 failed attempts on design and 3 different implementations that never finished. Fortunately for me, since I had a lot of failed designs (as opposed to implementations) I better appreciate existing designs even with their warts.
Well, guess what, it didn't actually turn out that way. At least I'm more understanding of people who annoyingly promote their own languages all the time.
What's worse is that PL authors are usually students of PL history, and the history of PLs is littered with "We did things this way out of necessity. We wouldn't have done this if we had more time/resources." Which would be fine but then we are all stuck using those tools with all their arbitrary limitations and quirks for 50 years. If you ever thought "I could write something to make this so much easier", the impulse is 100x worse as a PL designer, because you realize how many things we do are done for no reason at all beyond history and happenstance.
If not, your language will be marked and swept.
It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free, leaving you more time to work on designing the language to suit your needs. If you combine macros with clever uses of GADTs you can even get the host language to "type-check" programs written in your DSL. OCaml and Lisp/Racket are my favourite host languages to use for this approach.
It is still a fun exercise, though.
- a complete parser/compiler and debugger toolchain
- a complete language server implementation, and language modes for all popular IDEs including VSCode, VIM, Emacs, etc.
- a package manager with a large, secure, and vibrant package ecosystem
- a community with live chat support and a strong Stack Overflow presence
- robust documentation with copious examples, tutorials, guides, and technical info. These days even a multi-hundred page book with professional editing available for free is expected.
- That you will be responsive to bugs, provide patches promptly, and review/merge community feature requests asap. They will get very mad with you if you don't, and likely write a blog post about how much of a tyrant you are over your project.
- your language is expected to have undergone a full independent security audit, and to be up-to-date on the latest security issues. It's expected to be stable with complete backwards compatibility.
- you have to manage the community and deal with petty interpersonal conflicts or else word gets out that your language has a "toxic community"
- support for all major operating systems and hardware architectures including Mac, Windows, and all distributions of Linux, as well as esoteric hardware.
- support for the web through wasm, so you need experts at that as well as x86 and ARM platforms now.
- And on top of all that, your project needs to be completely open source, with a free and open license that permits royalty free, patent free, commercial use. Oh, and your users both corporate and personal expect your work to be completely free as in beer. They won't pay anything for it. Not even a dollar. And if you ask them to pay something they don't weigh the value of your product versus the competition, they balk immediately and don't even consider it. It's been a non-starter to ask for money for a PL for decades.
It used to be the case that you could write down a spec on paper and call it a PL. Back in the 60s you could call that a "programming language" without even writing an implementation of the parser. Today a programming language is a massive undertaking and the saddest part of it all is that it's been commoditized. I've even seen a sentiment out there that programming languages are done, that we've invented all we need to. All future languages are unnecessary, and all current languages are largely the same. Could you imagine? So as hard as programming languages are, DO write them. That's my message. Just do so responsibly and with clear eyes and expectations.
Regarding language innovation, I will oversimplify and assert that due the resistance to modernization (for whatever reasons), we have spent the last decades making Algol 68, Lisp and ML appealing to mainstream, smuggling their capabilities into more acceptable packaging.
You have https://microsoft.github.io/language-server-protocol/ for doing the integration between IDE and your parser.
Also, https://highlightjs.org/ which lets you broadcast your code in a pretty way.
I do however feel that your QoL will depend on whether or not you make a compiler or a translator. The key benefit of a translator is that you can co-opt the host language's standard runtime and debugger.
I don't understand what you mean by this, so I'm hoping you can clarify. What's the advantage of doing this over the alternative way: writing a parser which would compile to code in the host language, or some primitives that you've arranged in the host language that would represent the bytecode of your language interpreter. From there you can leverage whatever in the host language you want, including its type system, packages, language semantics, FFI access to other languages. Really whatever you like.
What you're suggesting is leveraging a macro system to help you write what is essentially a parser/compiler, which is a good win. But I don't see what that buys you past that point. If I do things the other way, by writing a parser that consumes a *.my_lang file and compiles it to primitives or actual source code in the host language, I can still leverage all the benefits of the host language just as you describe doing with a macro system. The advantage of doing things this way is you're not limited to languages with tolerable metaprogramming capabilities.
Less tangentially, I've worked for the past 8 years on building a HTML5 canvas library in Javascript[2]. I don't ever see people talking about JS libraries in terms of computer language design but I think, in a way, that is effectively what a (good) library is doing: building new, sometimes more developer-friendly, ways of conceptualising and creating programs that do stuff using novel APIs, syntax, state, etc.
There's a lot in the OP that resonates with my experience:
- "once you design a programming language you fall down a rabbit hole" - I've lost 8 years of my life to my project; but I've also gained 8 years of learning skills and insights that do translate to my everyday work as a frontend developer. For instance, almost all of my knowledge about accessibility comes from trying out many (many) different approaches to making the canvas element more accessible.
- "You keep adding new features that are really cool" - Guilty! My library doesn't need an integrated reaction-diffusion engine, but somehow it's acquired one along the way. Similar story with the reduced palette filter which, while a lot of fun to build and play with, remains too slow for use in production eg dithering live-stream video.
- "It also affects your communication with the rest of the programming community" - Because of the need for speedy calculations, I've developed an approach to programming Javascript which tends to avoid slower functionality. This sometimes leads to me writing code that could/should be written in a clearer, more concise and understandable way: 99% of JS code doesn't need to be super-fast, just fast enough!
[1] - for example, this one: https://gevey.rikweb.org.uk/
[2] - Scrawl-canvas - https://github.com/KaliedaRik/Scrawl-canvas
I implemented it with a parser that I developed myself. So, actually, I fell in the trap of developing a language myself, not a programming language, but a developing my own language developing tools. And I have doing it for more than 20 years now, on and off. I do recognize the self promoting bit. (I am kind of doing it right now.)
Why I did it: in 2013-2014 I was SUPER bored and burned-out, I needed a challenge and I picked that one. Oh, I was wrong.
The GOOD: the challenge has been fun, I learned A LOT. I learned how to use ANTLR, and then I learned that ANTLR was getting in the way and now I'm wary of ever using a parser generator at all, for sure not that one. I learned how to write a recursive descent parser, even if I did it by reversing engineering (i.e. mostly copying) the original rd-parser of puc Lua. And I learned that no matter what I do, what I find the hardest is respecting the correct operator priority in math expressions. I know that shouldn't be the hardest thing, but brains are fun. It also ended up getting used. It was used in the OnePassword client for windows, and it's for sure used in the Shenzen IO and TIS-100 videogames. I'm sure there are many more users using that. It's stuck somewhat in between of "nobody uses it" and "it has enough traction that the community will take it forward and you can forget it even existed", and that, turns out, is not a great place to be.
The BAD: I picked C#, as that's the language I was using at the time and I had an interest in seeing it used in Unity. Big mistake. Don't get me wrong: C# is a great language and if you're planning to base your company on it, I don't see big reasons not to (I wouldn't, probably, but every language has its pros and cons). But C#, and in 2014 to boot, was a mistake for such a project. First, however multiplatform C# became, a lot of development is still Windows only (at least, if you want to target the main frameworks). And second, 2014 is precisely the year when C# started changing radically with the core / standard frameworks. The situation now is probably more stable, but in 2014/2015 it was a mess, and half of the reflection frameworks were broken to say the least.
The UGLY: I just didn't consider how much punches life can hit your face with. I changed job, twice. Changed platform from Windows to MacOS, to Linux, to MacOS again, now I program for multiple platforms in Rust most of the time, and my interest in C# is close to non-existent. Family members passed away, new kids were born, situations changed, and life hit me hard enough. Energy got focused on daily jobs, or on other side projects. Every other side project, by the way, has been met with humongous amounts of guilt, because I'm/was leaving moonsharp users (however few they might be) behind.
So where I'm now? I tried to delegate the project to other people, but I'm not effective at it (maybe they are, but in reality, I'm just absent). I tried to get back active into the project but failed a couple of times and now I fear doing it again and creating expectations. Sometimes I feel like contributing something, but I can't even get a proper dev setup running. I don't remember details well enough to not break something when fixing major bugs, nothing major but usually it goes over the amount of energy that I would like to spend. Sometimes I read some question on MoonSharp, and I try to answer, but more often I read a question and before answering I have a guilt trip because I'm answering to this question and left alone all the others and so I simply avoid this one too.
Most of all, the project is not aligned anymore to my interests, so it feels like a chore.
So, if you are reading this: think twice before making the same mistake. If you want a side project, pick one that is either small, or has a clear end to it.
Beyond that, I tend to reach for:
- A config file. Can the problem be solved declaratively through a config file that a program reads and bases its behavior on?
- Templates, generators, and macros. Along the lines of syntax, would templates or generators solve problems with an existing GPPL?
- A domain-specific language. Does the problem lend itself to composable patterns or statements? Could a DSL help organize and simplify them?
- A transpiler. Is the syntax of an existing GPPL the problem? Would syntactic sugar be enough?
Of course, there are plenty of other reasons to write a new GPPL, like for your own learning or just as a hobby. Just know what you're getting yourself into.
* If the configuration file had mode 0755 then execute it, and parse the output.
* Otherwise read as-is.
That was my solution to writing a complex configuration file, which behaved differently upon different hosts - write the thing in perl, and have it "print" blocks and configuration values appropriately.
Jeff Duntemann
Loved Jeff's book on x86 assembly back in the day. I still remember him (in the book) introducing a DOS interrupt that I could invoke I think through DEBUG that rebooted the computer. And in the text he says something like that: that's the feeling of power. And he was right, that was exactly how I felt.
(Warning - comic gold, get a tissue) https://aphyr.com/posts/353-rewriting-the-technical-intervie...
And then read the rest of the stories on the site.
They're all both extremely technically interesting and very good short fiction.
The same goes for an OS or a game engine or any other large scope (Or collection of small scope) tool.
The other people who kinda care are those similarly engrossed in their own project, meaning... they do not actually have time to care.
My first priority with computing is to get rid of all the custom stuff. I won't even think about refactoring or any other improvement till I've weeded out all the handwritten reimplementations of library functions, made sure that there's no ad hoc string.split() based data formats that should be JSON, or UDP packets that should be MQTT.
Honestly, I don't even like that macros exist. I understand that in low level languages sometimes one needs metaprogramming, but it's more of an unfortunate reality than a cool feature.
Yes, like most initiatives, you are set for failure, if you define success by how popular a creation of yours becomes. You can still learn a lot.
And still, because "success" is such a rare breed, we actually need more people to do challenging stuff, so we'd get more success in absolute numbers.
The OP actually lacks sanity in recognizing that their project did not take off, and they don't have a clear idea of why they are doing their programming language project.
Basically, they are realising it's not a success, and they don't necessarily care about learning much out of it, and yet are still not letting go. If anything, that's a definition of insanity :)
Real insanity would be still thinking it was gonna be the next big hit.
I think we'd get more successes if people didn't didn't do this stuff and instead joined existing projects, or started with the business stuff first and tried to start their own foundations and such.
There are big wins, but they're incredibly rare and seem very random. And many of them are almost guaranteed to fail.
You'll learn a lot, but what can you actually use that knowledge for? Niche DIY stuff doesn't teach you much about mainstream methods.
The result is 1000 garage tinkerers for every Guido Van Rossum, and sometimes it's even actively detrimental to a project to try to cater to the tinkerers.
I think that you're working from a very different definition of "success" than your parent comment. "Success" doesn't have to mean "contributed to GDP". For some people "success" means "I had fun, learned stuff, and expanded myself as a person".
Not so much contributed to the GDP, but "Advanced the state of the art in some way" or "Solved a problem of interest outside the programming community".
I would imagine only the very most dedicated would actually enjoy doing a new language, maybe a third of programmers, and probably only those who fully accepted their thing was probably not going to be an enterprise grade solution.
I've met a bunch of people who do things like write their own typed concatenative language. So I can tell you, first of all, that they tend to spend the week merrily contributing to, say, orchestrating Kube clusters, and second, the time they spend bit banging their pet programming language wasn't going to go towards more orchestrating Kube clusters. Or whatever it is you think is actually important.
It might even be aimed at people deciding whether their team at work should pursue a new language for some specific task.
Maybe this could be solved with a special badge that tells people which of the Three Tribes of Programmers you're writing for...
For purely toy projects, I'm not sure if any code blog advice at all is that helpful, except possibly "Make stuff really small if you want to impress other programmers".
Best Practices don't apply the same way to this kind of thing, since in a real industry project the best practice would probably be "Just don't".
No language needs an IDE.
It is a very good experience, there is a lot to learn. As long as your goal is not to get it done quickly...
PL design is like crack for programmers. It is a lot of fun, which is why you see hundreds of little languages posted here every month.
The danger is that some people get don’t have a clear idea. About what it takes to write a pl. They can get quick wins with a parser generator and maybe some easy semantics, which give a good taste of how fun PLs are but also belie the fact that the hardest parts are exponentially harder than that.
Anyone embarking on a PL needs to be clear eyed about a few things.
1. You’ll never make money off of it. No one sells compilers anymore, and there are like 2 big for-profit languages out there out of thousands. If you do make money, let us all know how.
2. The odds are your language will have exactly one user, yourself. Get comfortable with the possibility you will spend years on something and no one will really appreciate it.
3. The real work in creating a language is not technical but community building. Languages need people who speak/write it to thrive, so as much or more work needs to be put into the community effort for the language to succeed. If all you want to do is work on technical problems that’s fine, but technical prowess doesn’t make a language popular.
4. Get ready for criticism. Every programmer has their opinions about languages and they will be sure to tell you their opinion of your language. It will not be good. The only people who will appreciate your language are other language designers. Most everyone else will start by telling you how you’re doing things the wrong way, and they’ll end by asking why you’re eating your time with all this anyway. Get ready for that and steel yourself against it.
Please, do not ever preach someone that if you cannot do something.
First, it's technically very difficult, but you will gain deeper insight into the art of the craft. So, if you are a TC chaser or career minded person, then spending half a year writing a language will help you master the coding aspect of the game. I've started many languages since I started college, and each one was instructional. (I'm now 40 and an early retiree)
Second, it's fun.
Third, it may turn into something new. If some people don't write a new programming language, then we are stuck with what we have. This advice basically admits that the status quo is good enough.
The authors saying that the language, as a project, is a lifetime appointment? Well, this reveals everything. I believe if you want to do a programming language, then you must be willing to invest at least a decade or two.
So, here, I am at forty preparing to launch a SaaS around a language that I designed ( http://www.adama-lang.org/ ). The kicker, I believe, is that a project like this requires wandering the desert alone for quite a while.
I'm preparing to launch, and I just started to load test my shiny new production cluster. Low and behold, it sucks. Fortunately, I have a tremendous number of dashboards and isolated it to how I'm interacting with RDS. I've got my work cut out for me which I'll write about.
However, I have a potentially interesting business precisely because I evolved a language which solved a niche use-case. The number of problems that I have had to solve up to this point is not for the faint of heart. Life and reality are harsh mistresses.
So, maybe, yes, you can save yourself some heartache by not writing a language. Perhaps, a better way to think or phase this is "Writing a programming language is a lonely affair that will most likely end in tragedy after a long death march".
From http://lambda-the-ultimate.org/node/5569 - Three Things I Wish I Knew When I Started Designing Languages. Alvaro's talk led to an HN discussion at https://news.ycombinator.com/item?id=19428433
Sounds like someone should persuade some big-name university to hand out honorary PhDs to people who build new languages that produce demonstrable value (even if the value is not business value).
Seems to be the criteria for a PhD, anyway.
My heuristic is that the difference in work required is around three orders of magnitude between creation, usefulness and maintenance.
That being said, I think every programmer would benefit greatly, just by attempting to write a language.
However, it is not as hard as the author suggests. First, there are excellent tutorials, textbooks (Wirth, Dragon book) and tools available (lex, yacc, LLVM, CUP, Jflex, Lemon, ...).
You should give it a try just for the fun of learning.
You will not often _need_ to do it, as most capabilities can perhaps be realized as libraries (including network and mobile functionality mentioned by the author).
Once you design something useful, other people will likely chime in and advertise for your, write documentation etc.
Most complex software system incorporate compiler techniques or Domain Specific Languages (DSLs), for parsing configuration files or as a basis for building higher-level functionality (AutoLISP -> AutoCAD, ELISP -> Emacs).
He shouts out to Nanopass, a way of writing highly compositional compilers developed in the Scheme world and much ported elsewhere. This kind of approach much reduces the pain of designing and maintaining compilers and can lead to very efficient compilers: it is the technology used in Chez Scheme. It's not got much take-up outside of the Scheme-aware world, despite the existence of many ports of the essential idea to other languages; I note that interesting Nanopass links keep getting submitted to HN without getting commented on: https://hn.algolia.com/?q=nanopass