The website seems to have a bug with syntax highlighting. Pieces of code included in the post text are black, you can still see the actual text if you select it with your mouse. Same bug on Chrome desktop and on Safari on iPad
This is the classic example of why you should always set the foreground color when changing the background color. Otherwise you end up with <code> blocks that are black-on-black. It probably works fine if your browser is in dark mode.
A workaround is to open one of the code elements in the browser's debugger and add color: white to the :not(pre) > code style.
I must admit - I still don't understand macros. I get that they're code that's generated at compile time. But I don't understand how that's different than a function which evaluates other functions. I guess the latter would actually be evaluated at runtime? I think I get it conceptually but I'm not sure I have the muscle memory to reach for them. Anybody here have an "ah hah!" Moment with macros?
The arguments presented to a macro don't have to be valid code. Your "function evaluating functions" are all individually valid functions you've composed at runtime.
(defmacro foo [code] ,,,)
(foo "<some totally different language in a string>")
=> actual, compiled Clojure function built up by `foo` parsing the string, producing a Clojure list, and calling `(eval the-list)` and returning it.
I passed a string to `foo` in the example, but it could actually be anything the Clojure reader can parse even if it is semantically invalid—arbitrary Clojure data.
Many problems can be solved with either macros or higher-order-functions. The advantage of macros is two-fold (macros also have disadvantages, and Lisp allows you to use either solution; deciding which to use is often a matter of taste):
1. Syntax can be more familiar
2. Performance (to the extent that there is overhead for functions-calling-functions).
For example, let's consider a hypothetical lispy language that doesn't have a short-circuiting "and" operator. Macros would let you implement something to used like this:
(and (foo x) (bar y))
Higher order functions would require you to do something more like:
(and (lambda () (foo x) (lambda () (bar y))
More noise, and more work for the optimizer (or in the worst-case, more work at run-time). Languages that rely heavily on higher-order-functions will tend to have terser syntax for anonymous functions. For example in javascript you might do:
> But I don't understand how that's different than a function which evaluates other functions.
You want a function that takes in arguments, but does not evaluate the arguments when called. So:
func(foobar(), foobar())
Normally, foobar() will be called twice - at the time of the call. With macro expansion, you can ensure that it's not the result of calling foobar that goes into the func, but this expression.
A canonical example is if you want to write an if/then/else function:
if(condition, then_path, else_path)
It's quite possible that the else_path is invalid and will terminate the program if the condition is true. But if you wrote a function this way, it will evaluate both then_path and else_path - not something you want to do in a regular if expression!
function can do what a macro does, but it must run every single time while the macro can run just one time during the final code generation.
Let's say you have some Java-style a + b. It needs to work on native strings (concatenation) along with various kinds of ints and floats. The issue is that each one of these works differently internally, so the system is going to look at the incoming symbols and metasymbols (inferred type data) and is going to dynamically change between different functions like ADD_STR, ADD_INT8, ADD_INT32, ADD_FLOAT32, etc. If it did this at runtime by introspecting the type with some `switch(incomingType)` statement which involves all kinds of extra data and branches that clog the cache and create branches (two of the worst things you can do for performance). Instead, the macro (though they probably don't call it that) looks at everything and hardwires the correct output.
This hardwiring and its associated performance is the difference between a function and a macro. A macro `foo!(a b)` can inline into the current function while the function alternative must save all the registers to the stack and create a new stack frame which is very expensive.
The beauty of lisp macros is how functions and macros look the same. You may think that `(+ a b)` is the same as your favorite languages' `a + b`, but most lisps also allow `(+ a b c)` too. `+` is a macro rather than a function and it knows that `(+ a b c)` needs to convert to something more like `(+ a (+ b c))` and a typed lisp may go further to something like `(+f64 a (+f64 b c))`.
I think the best explanation, how Lisp macros are different from most other macro system is, that a Lisp macro is a function executed during compilation. It gets fed the parameters passed to it in literal form and then computes the source to actually compile by the compiler.
The power of this is that this function is also written in plain Lisp and it also sees all definition already existing in your Lisp image. Also, it is completely unlimited in power. So while being run, or as it is called "during macro expansion" the function could google for one of the things passed and use the result to create the output. Of course, this is not a practical example, but shows how flexible the system is.
There are some truly powerful and unique things about Lisps, but I wish articles like this would stop including REPLs and hot-reloading. The former have been table stakes for interpreted languages (and some compiled ones!) for years, and the latter is neither unique nor particularly widely used (hot reloads have to tangle with state and patching, so resetting the world for ease of reasoning is considered a best practice for a reason).
Not all Lisp dialects have well-developed hot patching systems. The OOP system has to be carefully designed for it. What if a class definition is superseded by a reload, but there are existing instances? The Common Lisp dialect of Lisp has useful answers to questions like this, but not necessarily every Lisp you come across.
Even Bash supports hot-reloading. I've developed modules that are updated in place by sourcing:
> and the latter is neither unique nor particularly widely used (hot reloads have to tangle with state and patching, so resetting the world for ease of reasoning is considered a best practice for a reason).
For what it's worth, hot reload is very widely used in Dart/Flutter. When you are writing UI code using a reactive style framework where rendering the UI appears to be "generate a new UI from scratch on each frame", it's more straightforward to have an intuition about what does and doesn't get reloaded.
It's not perfect, of course. But it works really well for the kind of changes you make when iterating on a user experience.
Generally speaking, the pain of handling hot patching depends highly on the structure of your codebase and the reason for hot patching to begin with. For a structure-of-arrays architecture, or for fixing logic errors at run-time, it's really no big deal at all.
It is limited utility, but if you start out from the mindset it's something you'll use a lot, those limitations start disappearing rather quickly.
Programming is in tension between the Light Side and the Dark Side.
The Light Side is about preventing the programmer from making mistakes: Get rid of go-tos! Add static types! Do not allow a bug to be expressible. The Dark Side is about giving power to the programmer: Macros? Obviously. Operator overloading? Self-modifying code? Multi-line reg-exps? Go to town!
The Light Side knows programmers are flawed and imposes constraints. The Dark Side trusts programmers with power. Neither side is correct all of the time, and a good programmer learns both.
Lisp is interesting in that it is clearly Dark Side programming (the programmer can do anything) but it's still admired by Light Side programmers. Maybe there's something about the simplicity of the language that makes it seem platonic--almost incorruptible. Or maybe Lisp is so pure that it embodies both Light Side and Dark Side, like a god that spawned the programming universe.
for a while there's been a strong cultural tendency in programming to mistrust people and trust tools (often even blindly because tools are just assumed to make no mistakes), and so expressiveness has been sacrificed for safety. There can be something to this but it's also self-fulfilling, if you strip people of agency of course they'll unlearn to program.
I've always thought it's a misanthropic philosophy and sucked much joy out of programming but also there's something to be said that there's safety in the expressiveness of Lisp. I read an article a while ago that found a robust correlation simply between length of a code base and errors regardless of languages used. And given how succinct and clear Lisp codebases can be that's valuable in itself.
we're kind of at the logical endpoint of this now with gigantic slop codebases that nobody understands just held together by 20 different tools, and if you ask me if I had to pick one of those or something one tenth the size written by a guy or girl who has been writing Lisp for ten years I'd say thank you I'll go with #2
I think the big thing is that Lisp is (aside from mutable variable assignment) basically all declarative, rather than the imperative paradigm.
Even without static types, and even allowing macro craziness, there's just such a stronger baseline of declarative and functional thinking, you're off to such a good start in clearer thinking and reasoning about a program.
It isn't nearly as much of a trade-off as people say. Languages like Haskell are both remarkably expressive and provide a lot of safety. (And, of course, languages like Python, Java and Go are the opposite.)
I think of C as more of a dark side language because it's basically a portable assembler. Lisp at least prevents you from doing the stupidest things like overrunning array bounds or writing to freed pointers*, which C is perfectly happy to let you do.
* Of course there are no user-visible pointers** in Lisp so the concept of free vs. not free doesn't come up because memory allocation and garbage collection are automatic.
**Under the hood most everything in Lisp is a pointer, but the user cannot see them or mess with them.
Lisp generally has precise GC, which I'd say makes it light side. It's even relatively type-safe if you count runtime type-checking.
Highly reliable systems are written in Erlang, which if you squint is another Lisp dialect. There's even a sexp-based version called LFE, for Lisp-flavored Erlang. Erlang's key to reliability is error recovery, rather than exceptional levels of error prevention.
I do like your light side/dark side classification.
I dislike when people use the term “power” to describe making code slightly shorter to type. Operator overloading is not in any meaningful sense more powerful than a language which lacks it.
That is a very powerful idea, but unfortunately it cannot be realized with a fixed set of language rules. Some invariants are universal, but some are bound to the domain. Some projects require, say, an int (EvenInt) to take on only even numbers and others odds (OddInt). You can imagine the possibilities here are unbounded ("the numbers should be prefixed by numbers that are divisible by my age squared").
Ideally your base language has the expressive power to formulate new abstractions and constraints. This fundamentally requires the Dark Side. Once the proper primitives are in place a different "language" - this can be literally or figuratively - is used to express the interplay between those primitives.
In effect you are starting with the most general (say, all of Lisp), restrict it to become more and more and specific - only "these modules" - until it cannot be reduced any further. If you can bring your domain down into being expressible as, say, a single config file, that'd be quite ideal. If you don't need the powers of abstraction to express your solution then exposing said powers would only invite in trouble. The mathematical equivalent of introducing unnecessary variables.
As a programmer you are free to traverse the journey from say all of Lisp to a JSON config file in whatever way you please.
The "Light Side" people have converged, or try to converge, on some intermediate state between full powers of abstraction and configuration only. I think this is useful because many if not all problems travel through that intermediate landscape on the way "down" (into their specificity). For example what you call "types" is a significant restriction on your freedom but at the same time it's so enormously general that this restriction can be considered a worthwhile default because just about any problem I can think of can potentially benefit from that restriction.
All this is to say that I don't think it's a dichotomy so much as two interacting polarities whose interplay gives each its strength.
Lisp is a particularly minimal base introducing very few restrictions of its own. But it's not the only one. Forth would be one in another direction.
But there are two types of “programming”: 1) programming per se (just you writing code for fun and perhaps building your own tools) and 2) a team of people with real-world constraints writing software for others for revenue
It’s clear why it feels so good to use something like Lisp for personal projects. It’s also clear why the vast majority of companies don’t use it
I'll paraphrase someone who commented on HN once (about dynamic vs static typing, IIRC): permissive languages are enablers for solo programmers, but you need more restrictive languages for team programming.
This matches my experience because you don't chose who you work with. Skill level is uneven among the team. Less skilled co-workers will make mistakes that will have more consequences when the language is more permissive.
That's one of the reasons you want code reviews, but then you convert your skilled programmers to teachers. This is not a good trade, because the time they spend improving the skills of other programmers is partially wasted because programmers are free to come and go. AI could change this picture.
The name of the game is therefore to keep your teams small and skilled, but this is can be difficult because of typical company politics - e.g. "bus factor" considerations, shortening time-to-market is often achieved by adding more programmers.
> Less skilled co-workers will make mistakes that will have more consequences when the language is more permissive.
Even the most skilled engineers are going to make these mistakes sometimes. That's the whole point of more restrictive tooling, and why Rust has exploded.
Don't get me wrong. I love Lisp, and use it in some personal projects. But I wouldn't want to use it in must not fail scenarios unless those situations can wait for a human to fix the running image.
I agree with you. Though I wonder if the divide is more about the size of the program. I've got a couple of large (>100K LoC) solo projects and I like using C++ because it is much easier to catch bugs at compile time.
Static typing is a huge advantage in refactoring, and if you don't refactor a large program as it grows, you end up with a mess.
If the more skilled programmers become managers, then it's theoretically their job to become teachers, as their goal is to be force multipliers as opposed to people who can't delegate and accept a lower quality (temporarily) in exchange for higher throughput later.
Programming is in tension between the Light Side and the Dark Side.
The Light Side believes in giving people freedom and power to achieve freedom, to allow themselves to express themselves better through programming, to become better programmers through the tools the light side provides; in the Light Side believes in programming as a way of thinking.
The Dark Side believes that people are incapable of handling freedom and must be restricted and controlled under the service of various organizations - generally companies. The companies will control access to programming and control the programmers via this access. The dark side does not believe in making programmers better programmers, it believes in managing programmers. The dark side does not believe in helping programmers to think, it believes in thinking for them.
You have it inverted. Your "Light Side" is what previous generations of programmers called "Bondage and Discipline", because this philosophy assumes that a good programming language has built-in shackles to impose the will of a so-called "benevolent dictator" on all programmers.
What side for python? Haskell? Idris? Clean? C? K?...
white/dark side is too much of a simplification to me.
Programming language equilibrium is a very complex subject and fascinating one.
After years of experimenting with many of them, it is all a matter of context of usage, personal preferences, objectives, etc.. there is no such thing as "the best programming language ever", neither "best for light side programming" nor "best for dark side programming".
Similarly, there is no such things as: Light side/Dark side in artistic painting. There are tons of paints some are clearly greater than others, but there is no such thing as "the best painting most beautiful for everyone seeing this paint comparatively to any pain".
Current SOTA models work great with Emacs Lisp and Common Lisp. Giving access to the REPL enables them to do an amazing job but they still work well with Unix input/output static code processing paradigm.
I use LLM_based coding agents frequently with Lisp languages and my way of working is different with Lisp languages: 1) if generated code ever has a syntax error, I like to quickly fix the syntax error myself. 2) for some reason I usually prefer to run tests myself in another terminal (with Python, Typescript, etc. I let the coding harness run tests).
I haven’t let coding harnesses run REPLs. When I do let a coding harness run tests I specify in, for examp,e, my Common Lisp skills file to run ‘sbcl -load …” so bash test commands are one liners.
It would be interesting to work on skills and a harness to use REPLs - nothing bad about that idea, I just haven’t tried it.
> [...] among its extensibility, its interactive environment, the REPL, and a lot of other features we haven’t touched yet. It is the combination of all of them that makes Lisp programming what it is.
Agreed. However, although the alternatives are few, they do exist. Today, I'd like to convince you (whether you're OP or a commenter) to give one of them a try. I'm talking about GToolkit[1]: Smalltalk/Pharo-based reimagining of Smalltalk as a productive environment for modern system design, analysis, and implementation. It's based on Pharo and its VM, but with GT-specific extensions and replacements, developed on GitHub in both Smalltalk and Rust.
I used both Common Lisp and Smalltalk over the years for some of my side projects. Technically, the environments are comparable: image-based, live, interactive development is central to both. Lisp is easier to fit into modern workflows: it's still just files on disk. If you ignore the REPL and treat the image as a bundler, you can have a Java- or Python-like development workflow. You can gradually adopt the more interactive ways of working with the codebase. Smalltalk was historically more of an "all or nothing" approach, but nowadays it supports Git-based workflows as first-class, with GToolkit providing additional tools directly. GT is tied to a single IDE, but that IDE is genuinely powerful, pragmatic, and easy to customize: the entire IDE is Smalltalk code[2] that lives in the image beside your code, so you can live-edit any part of it at any time.
The languages differ, most obviously in the object model (single inheritance and message sends in Smalltalk, and multiple inheritance with multimethods in CLOS), but I don't think one is strictly better than the other for the vast majority of code. You just need to structure your code differently. Both systems are very dynamic, so neither is like Java or C++. After quite a few projects in both, I am convinced that this difference doesn't matter.
Both languages have very simple syntax. Smalltalk doesn't have macros; instead, it has an extensible/replaceable compiler for method bodies. This capability is used, for example, for compiling grammars into parsers/lexers in SmaCC. It's not as convenient for control-flow abstraction as CL's macros, but Smalltalk tends to make them out of blocks (lexical closures with non-local exits) plus actions on thisContext. In practice, I never found either language too limiting in what I could express, and both have small, regular basic syntax.
TL;DR: Common Lisp and GToolkit Smalltalk offer comparable technical merits, with the most pronounced difference being GT's built-in, high-quality IDE, which open source Lisp lacks.
The IDE (and the capability to build cross-platform, but natively rendered GUI apps) is the major selling point of GToolkit, but the bigger reason to consider it is social. Lisp had a bit of a renaissance from 2005 to 2015, but ~it has since died down~ (EDIT: After checking a bit, it's more like there are similar levels of development now, but it's just varied, and I personally hit a few areas where nothing new happened for a while. Or in other words, the hopes of CL getting catapulted to mainstream by the wave of adoption didn't materialize, and the adoption seems to have largely plateaued, with a slight upward trend since then.) A lot of great things for Lisp appeared in that time: ASDF and Quicklisp being prime examples. Unfortunately, the momentum was lost, and since then, the rate of development has plummeted. It's still a solid proposition because of its stability, which means a lot of old code still works perfectly well, but it's a double-edged sword: it would take a hundred miracles in a row for CL to get a plausible M:N concurrency story, for example. This isn't true for Smalltalk, and doubly so.
There's a small but active community around Pharo, mostly academics and hobbyists (IME; and it's from a decade ...
All three major programming environments at Xerox PARC, shared similar concepts.
Interlisp-D, Smalltalk, Mesa (XDE) which evolved into Cedar.
If you read Xerox papers about all of them, there are several quotes on how relevant it was to share the same programming experience across environments.
Which is why, given their linage, JVM and CLR are the closest big mindshare ecosystems that somehow still have traces of those features when using their IDEs and runtimes, even without being a proper Smalltalk or Lisp.
There are many articles extolling the virtues of Lisp. I would like to see some articles that have a level-headed criticisms or critique of Lisp, it's ideas and it's place in the ecosystem of languages.
Articles like this, and the PG articles it references, amount to "if you know, you know". I understand the appeal and I understand the explicit and implicit arguments this article is making.
Computer programming has matured quite a bit in the past 60 years. I would like to see more articles that are more considered in their examination.
As a language Lisp is great (though the ecosystem is limited). It has two flaws-- it's very open-ended so unless you're talented and disciplined you can fall into a rabbit hole of hacking fun Lisp stuff and not actually getting any work done. Other languages have this problem, but Lisp I think more so.
The second flaw is that it eats perfectly capable minds for years and the results don't justify the time investment. Python or whatever is fine. I wish I took most (but not all) of the time I invested in Lisp and put it into something else instead.
> I would like to see some articles that have a level-headed criticisms or critique of Lisp, it's ideas and it's place in the ecosystem of languages.
Standardized Concurrency is basic table-stakes for a language today. CL does not have a standardized async/await or concurrency model. The standard hasn't been updated since 1995 so it will never happen.
Your wording sounds like it implies that Lisp "got stuck" somewhere in the past, no?
Clojure, Clojurescript, Clojure-Dart, Fennel, Jade, Jank, Jolt, Coalton - these are relatively recent (and still developing) languages, and this is just off the top of my head, there are so many more.
Lisp is not a programming language (in a sense), it's an idea. It influenced pretty much every single PL we use today and continues to do so. You can't really "level-heatedly" criticize an idea, it's like criticizing I dunno, group theory. You can though debate about merits of a specific implementation of it.
Certain people seem to find it mentally appealing. I would liken it to one of the really niche music genres. Only a small number of people like it, but they do so very intensely. The Stockhausen of programming.
That's basically all it is. A fandom. A remarkably enduring one, but not one which has ever broken out, and therefore is unlikely to ever do so. Now in an even worse position: if you are armed with an LLM and therefore uninterested in the code itself, why on earth would you direct it to write in Lisp?
I've been programming Lisp too long to appreciate it's features. It is not until I watch in shock and horror how people sabotage themselves in the popular and economically less risky languages that I suddenly come to appreciate Lisp again.
I played a little bit around with LISP but I was astounded when I saw that you run programs using the live REPL. I thought, how can you reliably deploy a finished program if it can get messed with at any time? Why isn't there a way to compile to a binary?
I’m guessing there’s some way for a Lisp program to load a DLL and call a function in it so writing a Windows application using the Windows App SDK is likely possible.
The one part I can’t imagine is how do you pass a Lisp function as a callback?
The inflection of the copula "can be" is exactly the problem, as is the 'anything computable'. There's a reason Isabelle is used as a proof assistant, and Prolog isn't. The former guarantees sound, mathematical rigor that can't be trivially violated. The latter doesn't. Prolog is much more powerful and flexible than Isabelle. Power and flexibility are disastrous for safety. As someone else said elsewhere in this thread, safety is about restrictions.
In this case, it actually runs deeper than this, there are structural issues which prevent them from being sound right from the beginning. It's the mere fact that the whole behavior of your DSL can be silently changed without touching the module it's defined in, or the file it's being imported into. A well-meaning junior, in a completely different department, can completely destroy your invariants in a completely unrelated part of the codebase. Whether by shadowing runtime functions, or a package-level collision that exists upstream. Even Scheme's hygienic macros don't actually solve this, they just make it less likely. At the point in which you have no behavioral guarantees of semantics, you do not have a safety mechanism at all.
> There is no 'evaluative indirection' because a macro is a _compiler_ and the underlying code can be extremely alien to the DSL semantics.
This is a trivial contradiction. That is precisely evaluative indirection. Yes, stacking multiple compilers on top of each other, especially when their semantics are non-isomorphic, is a massive burden of mental overhead. Given you already have this problem with the mapping between Lisp and binaries, adding even more complicated layers on top of it is a recipe for disaster. Particularly because: you are going to make mistakes in your logic. Macros provide no means to stop you from blowing your own leg off when writing them. That is not what they are designed for.
Finally, you cannot escape the awkwardness of the underlying metaprogramming system, you will always have to wrestle with quotation and quasiquotation. You have to define a DSL before you can use it.
Only at the point in which you treat and use Lisp like an unserious toy, rather than the powerful tool that it is, can the fantasy of "macros are a safety mechanism" begin to make sense.
It's really fun: I read so many propaganda on Lisp here that by now we should all be coding in it. Don't we get perhaps weekly posts like this?
But ... somehow almost everyone simply ignores this propaganda --- that's actually the way we ought to handle propaganda! Funny that it works with Lisp, but not with many political topics where it can be shown that propaganda really works.
My experience with Lisp is somewhat special-cornered. I only know Emacs-Lisp, and that not great. And for me, Lisp an out, outdated and confusing thing.
I don't know ANY other programming language that elected assembler mnemonics to be "high"-level expression: car and cdr.
While I love Emacs, they claim all the time that is has 100% discovery through build in doc. But that is only 70% true. It doesn't exist in Lisp like it does in other languages. For example, I have an association list. Now, what functions exist that uses them? I can't use the equivalent for (dir alist), alist.dir(), or use some LSP for that. I can't even look in the symbols, because operators are written like assoc or assq with not logic. And then there are either prefixes to them (rassoc) or postfixes (assoc-string).
That there is no namespace is soooo 1970. I mean, even Turbo-Pascal had namespaces.
Typechecking is nonexistant.
At least we have byte-compiling.
And note that I don't croak about the parenthesis. While I find it mildy weird that I as a human program in almost-AST, Emacs has superp support for the parenthesis. So people croaking about them just had never had a good editor IMHO.
Now, one can argue that Emacs Lisp != Common Lisp. But these propaganda articles don't tend to promote one specific Lisp, they only talk in generals. So I can apply this general principle and compare it to the tiny corner of these awful language family.
Don't miss out, we had new excellent editors and tools being released in the last months:
Mine: a complete, single-download application that comes with everything needed to experience the interactive and incremental development programming workflow, including hot-reloading and on-the-fly debugging. For CL and Coalton.
89 comments
[ 2.0 ms ] story [ 88.6 ms ] threadA workaround is to open one of the code elements in the browser's debugger and add color: white to the :not(pre) > code style.
I passed a string to `foo` in the example, but it could actually be anything the Clojure reader can parse even if it is semantically invalid—arbitrary Clojure data.
1. Syntax can be more familiar
2. Performance (to the extent that there is overhead for functions-calling-functions).
For example, let's consider a hypothetical lispy language that doesn't have a short-circuiting "and" operator. Macros would let you implement something to used like this:
Higher order functions would require you to do something more like: More noise, and more work for the optimizer (or in the worst-case, more work at run-time). Languages that rely heavily on higher-order-functions will tend to have terser syntax for anonymous functions. For example in javascript you might do:You want a function that takes in arguments, but does not evaluate the arguments when called. So:
Normally, foobar() will be called twice - at the time of the call. With macro expansion, you can ensure that it's not the result of calling foobar that goes into the func, but this expression.A canonical example is if you want to write an if/then/else function:
It's quite possible that the else_path is invalid and will terminate the program if the condition is true. But if you wrote a function this way, it will evaluate both then_path and else_path - not something you want to do in a regular if expression!Let's say you have some Java-style a + b. It needs to work on native strings (concatenation) along with various kinds of ints and floats. The issue is that each one of these works differently internally, so the system is going to look at the incoming symbols and metasymbols (inferred type data) and is going to dynamically change between different functions like ADD_STR, ADD_INT8, ADD_INT32, ADD_FLOAT32, etc. If it did this at runtime by introspecting the type with some `switch(incomingType)` statement which involves all kinds of extra data and branches that clog the cache and create branches (two of the worst things you can do for performance). Instead, the macro (though they probably don't call it that) looks at everything and hardwires the correct output.
This hardwiring and its associated performance is the difference between a function and a macro. A macro `foo!(a b)` can inline into the current function while the function alternative must save all the registers to the stack and create a new stack frame which is very expensive.
The beauty of lisp macros is how functions and macros look the same. You may think that `(+ a b)` is the same as your favorite languages' `a + b`, but most lisps also allow `(+ a b c)` too. `+` is a macro rather than a function and it knows that `(+ a b c)` needs to convert to something more like `(+ a (+ b c))` and a typed lisp may go further to something like `(+f64 a (+f64 b c))`.
The power of this is that this function is also written in plain Lisp and it also sees all definition already existing in your Lisp image. Also, it is completely unlimited in power. So while being run, or as it is called "during macro expansion" the function could google for one of the things passed and use the result to create the output. Of course, this is not a practical example, but shows how flexible the system is.
Even Bash supports hot-reloading. I've developed modules that are updated in place by sourcing:
For what it's worth, hot reload is very widely used in Dart/Flutter. When you are writing UI code using a reactive style framework where rendering the UI appears to be "generate a new UI from scratch on each frame", it's more straightforward to have an intuition about what does and doesn't get reloaded.
It's not perfect, of course. But it works really well for the kind of changes you make when iterating on a user experience.
It is limited utility, but if you start out from the mindset it's something you'll use a lot, those limitations start disappearing rather quickly.
The Light Side is about preventing the programmer from making mistakes: Get rid of go-tos! Add static types! Do not allow a bug to be expressible. The Dark Side is about giving power to the programmer: Macros? Obviously. Operator overloading? Self-modifying code? Multi-line reg-exps? Go to town!
The Light Side knows programmers are flawed and imposes constraints. The Dark Side trusts programmers with power. Neither side is correct all of the time, and a good programmer learns both.
Lisp is interesting in that it is clearly Dark Side programming (the programmer can do anything) but it's still admired by Light Side programmers. Maybe there's something about the simplicity of the language that makes it seem platonic--almost incorruptible. Or maybe Lisp is so pure that it embodies both Light Side and Dark Side, like a god that spawned the programming universe.
Relevant PG: https://paulgraham.com/langdes.html
I've always thought it's a misanthropic philosophy and sucked much joy out of programming but also there's something to be said that there's safety in the expressiveness of Lisp. I read an article a while ago that found a robust correlation simply between length of a code base and errors regardless of languages used. And given how succinct and clear Lisp codebases can be that's valuable in itself.
we're kind of at the logical endpoint of this now with gigantic slop codebases that nobody understands just held together by 20 different tools, and if you ask me if I had to pick one of those or something one tenth the size written by a guy or girl who has been writing Lisp for ten years I'd say thank you I'll go with #2
Even without static types, and even allowing macro craziness, there's just such a stronger baseline of declarative and functional thinking, you're off to such a good start in clearer thinking and reasoning about a program.
* Of course there are no user-visible pointers** in Lisp so the concept of free vs. not free doesn't come up because memory allocation and garbage collection are automatic.
**Under the hood most everything in Lisp is a pointer, but the user cannot see them or mess with them.
Highly reliable systems are written in Erlang, which if you squint is another Lisp dialect. There's even a sexp-based version called LFE, for Lisp-flavored Erlang. Erlang's key to reliability is error recovery, rather than exceptional levels of error prevention.
I do like your light side/dark side classification.
That is a very powerful idea, but unfortunately it cannot be realized with a fixed set of language rules. Some invariants are universal, but some are bound to the domain. Some projects require, say, an int (EvenInt) to take on only even numbers and others odds (OddInt). You can imagine the possibilities here are unbounded ("the numbers should be prefixed by numbers that are divisible by my age squared").
Ideally your base language has the expressive power to formulate new abstractions and constraints. This fundamentally requires the Dark Side. Once the proper primitives are in place a different "language" - this can be literally or figuratively - is used to express the interplay between those primitives.
In effect you are starting with the most general (say, all of Lisp), restrict it to become more and more and specific - only "these modules" - until it cannot be reduced any further. If you can bring your domain down into being expressible as, say, a single config file, that'd be quite ideal. If you don't need the powers of abstraction to express your solution then exposing said powers would only invite in trouble. The mathematical equivalent of introducing unnecessary variables.
As a programmer you are free to traverse the journey from say all of Lisp to a JSON config file in whatever way you please.
The "Light Side" people have converged, or try to converge, on some intermediate state between full powers of abstraction and configuration only. I think this is useful because many if not all problems travel through that intermediate landscape on the way "down" (into their specificity). For example what you call "types" is a significant restriction on your freedom but at the same time it's so enormously general that this restriction can be considered a worthwhile default because just about any problem I can think of can potentially benefit from that restriction.
All this is to say that I don't think it's a dichotomy so much as two interacting polarities whose interplay gives each its strength.
Lisp is a particularly minimal base introducing very few restrictions of its own. But it's not the only one. Forth would be one in another direction.
It’s clear why it feels so good to use something like Lisp for personal projects. It’s also clear why the vast majority of companies don’t use it
This matches my experience because you don't chose who you work with. Skill level is uneven among the team. Less skilled co-workers will make mistakes that will have more consequences when the language is more permissive.
That's one of the reasons you want code reviews, but then you convert your skilled programmers to teachers. This is not a good trade, because the time they spend improving the skills of other programmers is partially wasted because programmers are free to come and go. AI could change this picture.
The name of the game is therefore to keep your teams small and skilled, but this is can be difficult because of typical company politics - e.g. "bus factor" considerations, shortening time-to-market is often achieved by adding more programmers.
Even the most skilled engineers are going to make these mistakes sometimes. That's the whole point of more restrictive tooling, and why Rust has exploded.
Don't get me wrong. I love Lisp, and use it in some personal projects. But I wouldn't want to use it in must not fail scenarios unless those situations can wait for a human to fix the running image.
Static typing is a huge advantage in refactoring, and if you don't refactor a large program as it grows, you end up with a mess.
The Light Side believes in giving people freedom and power to achieve freedom, to allow themselves to express themselves better through programming, to become better programmers through the tools the light side provides; in the Light Side believes in programming as a way of thinking.
The Dark Side believes that people are incapable of handling freedom and must be restricted and controlled under the service of various organizations - generally companies. The companies will control access to programming and control the programmers via this access. The dark side does not believe in making programmers better programmers, it believes in managing programmers. The dark side does not believe in helping programmers to think, it believes in thinking for them.
What side for python? Haskell? Idris? Clean? C? K?...
white/dark side is too much of a simplification to me.
Programming language equilibrium is a very complex subject and fascinating one.
After years of experimenting with many of them, it is all a matter of context of usage, personal preferences, objectives, etc.. there is no such thing as "the best programming language ever", neither "best for light side programming" nor "best for dark side programming".
Similarly, there is no such things as: Light side/Dark side in artistic painting. There are tons of paints some are clearly greater than others, but there is no such thing as "the best painting most beautiful for everyone seeing this paint comparatively to any pain".
[^Malbolge]: https://en.wikipedia.org/wiki/Malbolge
[^Clean]: https://en.wikipedia.org/wiki/Clean_(programming_language)
[^K]: https://en.wikipedia.org/wiki/K_(programming_language)
I haven’t let coding harnesses run REPLs. When I do let a coding harness run tests I specify in, for examp,e, my Common Lisp skills file to run ‘sbcl -load …” so bash test commands are one liners.
It would be interesting to work on skills and a harness to use REPLs - nothing bad about that idea, I just haven’t tried it.
> [...] among its extensibility, its interactive environment, the REPL, and a lot of other features we haven’t touched yet. It is the combination of all of them that makes Lisp programming what it is.
Agreed. However, although the alternatives are few, they do exist. Today, I'd like to convince you (whether you're OP or a commenter) to give one of them a try. I'm talking about GToolkit[1]: Smalltalk/Pharo-based reimagining of Smalltalk as a productive environment for modern system design, analysis, and implementation. It's based on Pharo and its VM, but with GT-specific extensions and replacements, developed on GitHub in both Smalltalk and Rust.
I used both Common Lisp and Smalltalk over the years for some of my side projects. Technically, the environments are comparable: image-based, live, interactive development is central to both. Lisp is easier to fit into modern workflows: it's still just files on disk. If you ignore the REPL and treat the image as a bundler, you can have a Java- or Python-like development workflow. You can gradually adopt the more interactive ways of working with the codebase. Smalltalk was historically more of an "all or nothing" approach, but nowadays it supports Git-based workflows as first-class, with GToolkit providing additional tools directly. GT is tied to a single IDE, but that IDE is genuinely powerful, pragmatic, and easy to customize: the entire IDE is Smalltalk code[2] that lives in the image beside your code, so you can live-edit any part of it at any time.
The languages differ, most obviously in the object model (single inheritance and message sends in Smalltalk, and multiple inheritance with multimethods in CLOS), but I don't think one is strictly better than the other for the vast majority of code. You just need to structure your code differently. Both systems are very dynamic, so neither is like Java or C++. After quite a few projects in both, I am convinced that this difference doesn't matter.
Both languages have very simple syntax. Smalltalk doesn't have macros; instead, it has an extensible/replaceable compiler for method bodies. This capability is used, for example, for compiling grammars into parsers/lexers in SmaCC. It's not as convenient for control-flow abstraction as CL's macros, but Smalltalk tends to make them out of blocks (lexical closures with non-local exits) plus actions on thisContext. In practice, I never found either language too limiting in what I could express, and both have small, regular basic syntax.
TL;DR: Common Lisp and GToolkit Smalltalk offer comparable technical merits, with the most pronounced difference being GT's built-in, high-quality IDE, which open source Lisp lacks.
The IDE (and the capability to build cross-platform, but natively rendered GUI apps) is the major selling point of GToolkit, but the bigger reason to consider it is social. Lisp had a bit of a renaissance from 2005 to 2015, but ~it has since died down~ (EDIT: After checking a bit, it's more like there are similar levels of development now, but it's just varied, and I personally hit a few areas where nothing new happened for a while. Or in other words, the hopes of CL getting catapulted to mainstream by the wave of adoption didn't materialize, and the adoption seems to have largely plateaued, with a slight upward trend since then.) A lot of great things for Lisp appeared in that time: ASDF and Quicklisp being prime examples. Unfortunately, the momentum was lost, and since then, the rate of development has plummeted. It's still a solid proposition because of its stability, which means a lot of old code still works perfectly well, but it's a double-edged sword: it would take a hundred miracles in a row for CL to get a plausible M:N concurrency story, for example. This isn't true for Smalltalk, and doubly so.
There's a small but active community around Pharo, mostly academics and hobbyists (IME; and it's from a decade ...
Interlisp-D, Smalltalk, Mesa (XDE) which evolved into Cedar.
If you read Xerox papers about all of them, there are several quotes on how relevant it was to share the same programming experience across environments.
Which is why, given their linage, JVM and CLR are the closest big mindshare ecosystems that somehow still have traces of those features when using their IDEs and runtimes, even without being a proper Smalltalk or Lisp.
Articles like this, and the PG articles it references, amount to "if you know, you know". I understand the appeal and I understand the explicit and implicit arguments this article is making.
Computer programming has matured quite a bit in the past 60 years. I would like to see more articles that are more considered in their examination.
It's been discussed many times on HN.
The second flaw is that it eats perfectly capable minds for years and the results don't justify the time investment. Python or whatever is fine. I wish I took most (but not all) of the time I invested in Lisp and put it into something else instead.
Standardized Concurrency is basic table-stakes for a language today. CL does not have a standardized async/await or concurrency model. The standard hasn't been updated since 1995 so it will never happen.
Your wording sounds like it implies that Lisp "got stuck" somewhere in the past, no?
Clojure, Clojurescript, Clojure-Dart, Fennel, Jade, Jank, Jolt, Coalton - these are relatively recent (and still developing) languages, and this is just off the top of my head, there are so many more.
Lisp is not a programming language (in a sense), it's an idea. It influenced pretty much every single PL we use today and continues to do so. You can't really "level-heatedly" criticize an idea, it's like criticizing I dunno, group theory. You can though debate about merits of a specific implementation of it.
Certain people seem to find it mentally appealing. I would liken it to one of the really niche music genres. Only a small number of people like it, but they do so very intensely. The Stockhausen of programming.
That's basically all it is. A fandom. A remarkably enduring one, but not one which has ever broken out, and therefore is unlikely to ever do so. Now in an even worse position: if you are armed with an LLM and therefore uninterested in the code itself, why on earth would you direct it to write in Lisp?
It uses python for the interpreter.
The one part I can’t imagine is how do you pass a Lisp function as a callback?
In this case, it actually runs deeper than this, there are structural issues which prevent them from being sound right from the beginning. It's the mere fact that the whole behavior of your DSL can be silently changed without touching the module it's defined in, or the file it's being imported into. A well-meaning junior, in a completely different department, can completely destroy your invariants in a completely unrelated part of the codebase. Whether by shadowing runtime functions, or a package-level collision that exists upstream. Even Scheme's hygienic macros don't actually solve this, they just make it less likely. At the point in which you have no behavioral guarantees of semantics, you do not have a safety mechanism at all.
> There is no 'evaluative indirection' because a macro is a _compiler_ and the underlying code can be extremely alien to the DSL semantics.
This is a trivial contradiction. That is precisely evaluative indirection. Yes, stacking multiple compilers on top of each other, especially when their semantics are non-isomorphic, is a massive burden of mental overhead. Given you already have this problem with the mapping between Lisp and binaries, adding even more complicated layers on top of it is a recipe for disaster. Particularly because: you are going to make mistakes in your logic. Macros provide no means to stop you from blowing your own leg off when writing them. That is not what they are designed for.
Finally, you cannot escape the awkwardness of the underlying metaprogramming system, you will always have to wrestle with quotation and quasiquotation. You have to define a DSL before you can use it.
Only at the point in which you treat and use Lisp like an unserious toy, rather than the powerful tool that it is, can the fantasy of "macros are a safety mechanism" begin to make sense.
But ... somehow almost everyone simply ignores this propaganda --- that's actually the way we ought to handle propaganda! Funny that it works with Lisp, but not with many political topics where it can be shown that propaganda really works.
My experience with Lisp is somewhat special-cornered. I only know Emacs-Lisp, and that not great. And for me, Lisp an out, outdated and confusing thing.
I don't know ANY other programming language that elected assembler mnemonics to be "high"-level expression: car and cdr.
While I love Emacs, they claim all the time that is has 100% discovery through build in doc. But that is only 70% true. It doesn't exist in Lisp like it does in other languages. For example, I have an association list. Now, what functions exist that uses them? I can't use the equivalent for (dir alist), alist.dir(), or use some LSP for that. I can't even look in the symbols, because operators are written like assoc or assq with not logic. And then there are either prefixes to them (rassoc) or postfixes (assoc-string).
That there is no namespace is soooo 1970. I mean, even Turbo-Pascal had namespaces.
Typechecking is nonexistant.
At least we have byte-compiling.
And note that I don't croak about the parenthesis. While I find it mildy weird that I as a human program in almost-AST, Emacs has superp support for the parenthesis. So people croaking about them just had never had a good editor IMHO.
Now, one can argue that Emacs Lisp != Common Lisp. But these propaganda articles don't tend to promote one specific Lisp, they only talk in generals. So I can apply this general principle and compare it to the tiny corner of these awful language family.
I looked at lisp many times and it just doesn't do it for me.
Mine: a complete, single-download application that comes with everything needed to experience the interactive and incremental development programming workflow, including hot-reloading and on-the-fly debugging. For CL and Coalton.
https://coalton-lang.github.io/20260424-mine/
OLIVE: a new hand-made plugin for VSCode.
ICL: a new REPL for the terminal and the browser with advanced features.
as a bonus: a JupyterLite kernel based on JSCL that runs 100% in the browser.
find them all: https://lispcookbook.github.io/cl-cookbook/editor-support.ht...