Ask HN: 30y After 'On Lisp', PAIP etc., Is Lisp Still "Beating the Averages"?
Obviously this is asking for subjective opinions and experience among the crowd.
In one or another Arc (or Bel) intro essay, there's this notion that "a true hacker's-dream language" would be a new Lisp other than Common Lisp which is there accused of various suckages, and apparently also other than all the numerous Scheme variants (at least in the "no such corset of 'hygienic' macros, plz" sense). Perusing the various search results on 'CL vs Scheme / Racket' had me personally more convinced of CL for heavy-duty "real-world", non-"write-yourself-a-Foo" projects, though (wrt. perf / compilers / library ecosystem).
The last time I got hooked on pg's persuasive Lisp essays and got into it for a short while (cooking up an SSR templating engine somewhere in the PHP/ASP/Handlebars sense for Hunchentoot as a starter exercise) was in 2008. Real-world projects lured me away soon enough then and I didn't really miss it tbh. That short episode of first Lisping to me was just "kinda alright" rather than the-most-hackery-experience ever. Then ca. 2011 unleashed the novelties (for me) of WebGL, NodeJS, Golang and more, and those were enough "hackers delight" playgrounds for me for years to come.
But revisiting the essays (by now it might be nostalgia, like re-reading/viewing one's certain personal all-time / youth-time favourite novels, comics, movies once per decade or so, I guess =) I'm once again lured by the promise of "the language at the end of the power spectrum" against which every other one is just some shade of "Blub". Then again, I went into almost every language I ever got into from this sort of promising sense of power potential, including Haskell, C, and even Go (pre-generics) in its own ways of "power through simplicity, and then just code your code-gens for those rare meta-needs". (Zig, too, but though enjoyable, that kept changing itself up frequently enough back then to move me back to more settled grounds.)
You see the one sticky situation I keep ending up in is that I'm, in pg's words, always "painting myself into a corner" all too soon, hacking away iteratively and "dirtily" (in the make-it-work, then make-it-fast, then make-it-pretty/correct/maintainably-architected spirit — but I don't really get to the last part and only some of the 2nd if at all —we're talking green-field personal side projects here—, but anytime I start with the last part, it goes nowhere fast, nay slowly actually). I can paint myself into a corner in QBasic as well as in Haskell and anywhere in between (and lest I totally burn out my natural curiosity toward tech to learn to make things, I have to keep reminding myself that they probably aren't to be blamed as I'm the constant factor in all these experiences). And I keep picking up that "building stuff On Lisp" is maybe The One basis where that's either least possible / likely, or not a particularly impeding problem / hindrance due to "ever molding the tech basis / language itself around and toward the problem you're tackling".
So, what's your hot take on this question, your personal opinion, your experience report? In 2024, some ~30 years after On Lisp and some ~20 years after those essays, all the other languages have kept picking up nearly everything once-cutting-edge about Lisp, other than the "parensy pesky" that pg insists is Lisp's real power (convincingly so, at least from-a-distance, I might add), that whole melange of "sexpr (non)syntax / code-is-data / code-is-the-AST / read-time+compile-time+run-time macros in the same (non)language".
(And then, do we start purity-spiraling in a feve...
175 comments
[ 9.6 ms ] story [ 255 ms ] thread> And if so, _which_ Lisp and why?
Are there any other production-ready Lisps except of CL?
Whatever X Clojure can not directly answer, the ever expanding universe of solutions based on the JVM can.
If the OP wants to stop painting themselves into corners, write more code. All systems offer up lots of paint and lots of corners. Only experience in those systems can guide you out of them.
I’m a CL fan, so I’d turn to CL if a CL implementation will work for you. But if you’re looking for an s-expr with no excuses, it’s Clojure on the JVM. Though pretty sure GraalVM can turn even Clojure into native code if that’s a goal.
Sadly the .Net Clojure seems to be unmaintained, and there aren't obvious leading choices for the LLVM or Python worlds (Hy exists, is usable and interoperable, but doesn't have a clear value prop, maybe macros). C world has Racket I suppose. WASM has an S-Expression representation, but that alone does not a lisp make.
Is that CL, Clojure, a std Scheme or something? I really don’t have any issues jumping in either of these 3 even after a decade or more, but maybe if something obscure is used? And I wonder about the difference in naming and commenting as well then.
The cultists are all crying, "you suck at naming! you suck at comments!" And maybe I do! But those 12 people who will have to maintain it are going to suck even more. That's where a more "blub-like" language shines. It is enforcing a standard. And the cultists cry again, "i'm so smart i don't need blub to hold my hands!" Ok, if you say so... but where is your Minecraft (blub)? where is your Excel (blub)? where is your Facebook or your Google (blub)? maybe we can just accept that lisp is nice in some places, but not so nice for others?
And we are pro static types, it’s just easier to add them after I am done experimenting. Not upfront. And we do via various (backward compat) custom CL constructs and provers.
Because I find with any well designed and flexible program in a statically typed language, you end up with the same "tube of something" as you have in your dynamic language examples.
Take any large generic framework in rust, the types are extremely generic, dispatching on more generic traits. It's hardly a tag that says: "blood, to heart" unless it is highly coupled and working in one area.
Yet at some point, the template is instantiated--in the text!--and you can fill-in-the-blank by reading instead of having to do a runtime probe. For large systems that have a lot of preconditions, the "runtime probe" approach can be a thick fat time-waster (i.e. caressing the program into a state where something is even in the tube).
Because I find that in a good dynamic environment, it's easier to figure out how to use something complex when compared to trying to uncover how and why the types line up.
The absolute extreme being something like smalltalk, where you just guess how it should be used and then fix it in the debugger until it works.
No matter the tool, it will be very difficult to try out all code paths to figure out what can be in that tube. The difficulty scales with the size / age of the application.
Take your typical 20 years old enterprise application. Nobody from the original team is around. Whole generations of programmers worked on the project, each with their own favorite patterns / code style / level of diligence. There's a massive amount of features, half of which you don't even know about. Test coverage is spotty at best. You see some common function triggered from hundreds of places, it makes a huge difference in your ability to reason about it if you have types on the data being processed or not.
http://www.aaronsw.com/weblog/rewritingreddit
"The others knew Lisp (they wrote their whole site in it) and they knew Python (they rewrote their whole site in it) and yet they decided liked Python better for this project. The Python version had less code that ran faster and was far easier to read and maintain."
Turns out, not once does the author actually give a reason why Python was better than Lisp for Reddit. The only references to that connection at all were claims that some “X” (all among the set that are oft-refuted by Lisp programmers) wasn’t the reason!
It reads more like a PR justification (“oh, there’s not much difference really, we even asked unspecified engineers who agreed with the change, and our code is working better and better!”) than an actual explanation.
The decision to write web.py instead of web.lisp was not elaborated on, however.
As mentioned above, the closest point I see to him discussing that decision is saying that ‘Python uses objects to make frameworks somewhat like the syntactic constructs Lisp allows you to make’, ie ‘you can do the same things as Lisp less easily in Python, so that’s not an advantage of Lisp’. Which is both incoherent as a refutation of the referenced reason to keep Lisp, and not a reason for the change to Python.
"THE PYTHON VERSION had less code that ran faster and was far easier to read and maintain."
I come from the opposite side and find python syntax and style stifling.
for example he dosent make any claim about the python implementation being "far easier to read and maintain" (even by proxy by claiming that CL was hard to read/maintain), and honestly starts out by absolutely singing CL's praises
his reasoning seems to boil more down to the ecosystem surrounding Common Lisps libraries being poorer than pythons:
> If Lisp is so great, why did we stop using it? One of the biggest issues was the lack of widely used and tested libraries. Sure, there is a CL library for basically any task, but there is rarely more than one, and often the libraries are not widely used or well documented. Since we're building a site largely by standing on the shoulders of others, this made things a little tougher. There just aren't as many shoulders on which to stand.
[etc...]
> So why Python?
We were already familiar with Python. It's fast, development in Python is fast, and the code is clear. In most cases, the Lisp code translated very easily into Python. Lots of people have written web applications in Python, and there's plenty of code from which to learn. It's been fun so far, so we'll see where it takes us.
We are still in that world. I do game programming, and am convinced that if a game programmer isn't aware of their memory allocations, then their game will drop frames. This is true today, even at 60hz (and things are moving towards 144hz, 240hz, and beyond). Reusing buffers is essential.
I think that being aware of when and where your memory comes from is the foundation of performance. The discipline of handling memory carefully tends to naturally lead toward making more performant decisions on everything else.
Separately, my gut feeling is that in 2024, programming language runtime technology is advanced enough that manually reusing buffers isn't necessarily a good use of hours-spent-optimizing. To pick on Python in particular, it _does_ actually reuse buffers when they only have a single owner; more languages might get this in the future via the Perceus refcounting work?
But only in theory. The shit that I keep seeing of incomprehensible spaghetti code in C and C++ disproves that theory on a daily basis.
Typing being optional is not a problem. Like, if I'm looking at some function, and for some reason it's not clear what type of argument it's expecting, with one command I can pull up a cross-reference of other code that calls that function, and jump to it, and jump to the definition of the argument. If it's an object, I can jump to the definition of the class. This isn't really any more burdensome than jumping to the class in the first step, especially because in static langs I'll occasionally need to look at uses in context via cross-references anyway.
I don't get your dismissal of runtime probes. In large static systems, runtime probing is necessary for me to orient myself, the types don't really help. "How does the execution thread even get here?" is easily answered by sticking a break point there and running the thing, then you can inspect the stack. Common Lisp also has a very nifty "trace" function built-in, with a keyboard stroke I now get output every time the function is called along with what its arguments were and what its return values were. I can do this for any function, even built-ins, no instrumentation setup necessary or modifying the code or stopping/restarting the program. The full interactivity also means I can just jump to the definition even in some library-of-a-library code and if it pleases me rewrite it to add some prints or other info or just fix a bug and move that redefinition to my project. It's much easier for me to investigate a large Lisp system than a large Java system even though I've probably had more experience with the latter. Even grepping stuff is rather simple because of naming conventions.
Lisp definitely allows you to write code that is very easy to come back to and understand. I’ve written code like that, and I’ve seen code like that in eg certain sections of the SBCL internals, among many other projects.
However, there’s a reason we don’t write all our code in Python or Assembly. The fundamental semantic restrictions of one and the simple functionality of the other are both traded off against horrible scaling relative to a technical project’s complexity and scope.
If you write a project with complex stages/components in those kinds of languages (or mimic their styles in Lisp code, for that matter), either you have to split those components into parts that don’t make sense separately (making the code base significantly more complicated than eg an equivalent Lisp codebase), or you have to write extremely long and convoluted programs to accomplish conceptually simple functions (making the code base significantly more complicated than eg an equivalent Lisp codebase).
Similarly, if you use the Python or Assembly coding paradigms for large projects, you quickly get lost in a sea of individually simple-seeming code segments, unable to grok the connections between said segments just because of how many things need to be kept track of.
In contrast, for Lisp languages:
A) The default style used by Lisp programmers is highly functional, using many higher-order functions, using macros that solve their problem-domains without explicitly exposing the code doing the solving (objects also fall into this general role, btw, which is apropos given the Common Lisp Object System was defined purely with macros), and clearly delineating code / systems with mutating effects.
While this style is admittedly harder to grok than Python/Assembly for simple cases, it has almost no reduction in reading-complexity as code scale/complexity increases. Complexity gets naturally encapsulated and automated by functions/macros/objects (as opposed to either a restricted subset of those or simply the developer’s own head, which are the 2 approaches most other languages offer for complexity management) (algebraic type systems are a very cool exception to the prior note, but a pervasive object system (like CLOS) paired with macros allows you to make a type system (like Coalton) without feature/ergonomics loss).
B) Interactive development and ergonomic DSL creation are objective wins for complex or large programs; the former significantly reduces iteration time for both cases, while the latter makes it easy to just write code once that addresses the complexity/scale of the domain and then never deal with that complexity again (or at least not with the full complexity, even if you don’t understand the domain well enough to obviate the irrelevant complexity entirely in your DSL).
Sure, hot-reloading in a few other languages gives us most of the former (although things like CL conditions, debugging REPL-layers, access to object contents & stack-frame inputs (only addressed by some debuggers, and only in a restricted way), and some other features still lack equivalents) and other languages generally use objects alone to address (most instances, though not all, of) the latter (rather than Lisp’s mix of macros/objects/higher-order-functions) (we’re ignoring the text-templating macros in languages like C++ because everyone agrees they’re more trouble than they’re worth). But Lisp is still by far the most featureful and ergonomic offering of both of these facilities.
Lisp makes metaprogramming easy via macros. It could be argued meta programming can be done in any modern language, but because lisp’s code is a data structure, metaprogramming in lisp is as easy as changing a value in an array, hash, or dictionary (whatever data structure you’re familiar with, it’s that easy), as opposed to regexing strings, escaping correctly, and other finicky and error-prone hijinks other (non homoiconic) languages necessitate.
The other thing lisp does well is its REPL. I’m not so familiar with it but AFAIK it’s like an interpreter and debugger combined into one, giving a very nice developer experience.
Read: In Python, the input is read as strings of code, which are then parsed and compiled. In Lisp, the input is read as data structures (lists, symbols, etc.) due to its homoiconic nature (code is data). This allows for macros and syntax rules to be applied even before evaluation.
Eval - The evaluation phase involves interpreting the compiled code. Python’s evaluation is less interactive; changes in function definitions or classes often require the code to be re-run entirely, and the state can become inconsistent without careful management. Evaluation in Lisp happens directly on the read data structures. Functions, macros, and variables can be redefined interactively without restarting the REPL, maintaining a more fluid development process.
Print - The result of the evaluated Python code is printed out in a standard format. The printing is straightforward but lacks customization unless explicitly coded. In Lisp, results are printed in a way that preserves the structural representation of the data, facilitating easier inspection and manipulation in subsequent steps.
Loop - In Python, the main loop doesn’t offer much in terms of tooling or integration. Extending the REPL with custom behaviors or tools requires additional packages or integrations. The Lisp loop often includes advanced features like hooking into debugging tools, live editing, and runtime modifications, providing a richer development experience.
So, it really makes you ask: can you actually have a proper, "true" REPL in a non-Lisp language? So while it's fair to call them REPLs in a technical sense, they more like "interactive shells" and not REPLs. You really need a Lisp to have a proper REPL.
There is no comparison between the Clojure REPL and the Python REPL. Python's is just literally a shell that you connect to and can run commands in. The Clojure REPL is far closer to a Python Jupyter notebook (if you know about it) built into your editor - it's a long-running process to which you can send commands all the time. But because of the way lisp is structured, and because of the tooling, you can just write Python code in the file you would normally use, stand on one line, and hit a command to get it sent to the REPL and the output of it written in the editor.
It's... hard to explain how powerful this is, and how much better things could be if we had similar tooling in Python.
Note: I still much prefer Python for many reasons, but this one aspect is something that I also used to think (yeah, Python also has a REPL) and I realize now that I had no idea what power this really has in lisp-world.
So, let's take a minimal, simple example:
and the equivalent code in Clojure: In Clojure, you can send any expression fragment like a single symbol, list or even incomplete code and it will be evaluated according to its context. So, you can sent `true`, you can send `println` and you can even send `(when true println)` or `(when true (println))`, or even `(when true ,,,, (println))` (you can replace commas with line breaks if you want) and it will still be evaluated.In Python, you can send `True`, and you can send `print`, but you can't send `if True: print` (even with corrected indentation), because Reader would have hard time making sense of that.
In Python, you need to adhere strictly to its syntax, and partial expressions that don't form a complete statement won't be evaluated meaningfully. In contrast, Clojure's REPL can work seamlessly with code fragments because the underlying representation of code is just data structures that are always valid within the REPL context.
What makes it mind-blowingly cool is that you can send these expressions to a remote REPL without any transformation, serialization, or encoding, because it's not just code — it is data. In fact, some teams do just that. They'll keep an open socket on a running app, possibly performing somewhere in the cloud, so they can interact with it in real time. Putting aside the notion that this might not be the greatest idea, let's agree that this is a cool feature. The famous example of that is when something broke on Deep Space 1 (DS1) spacecraft in 1998, the team was able to fix it via REPL, see https://www.youtube.com/watch?v=_gZK0tW8EhQ - "The Remote Agent Experiment: Debugging Code from 60 Million Miles Away"
Sure, yes, Python is great, but when it comes to REPL stuff, you simply need a Lisp. It really is an amazing feeling when you can, for example, poke into any HTML element on a page dynamically, directly from your editor. Sure, you can do that with devtools in the browser, but it's not the same. You still have to switch to the browser; you can't type statements that are too long; and it's just not very convenient.
Very nice (now years old) example is when Bruce Hauman showcased developing a "flappy-bird" clone - he would tweak the params, or straight change the code in his editor and the bird in the browser would fly differently, it's literally like writing code while playing a video game. Or Sam Aaron, writing music with Overtone is another great example. Another, more practical example is when you're exploring API response results, if you have REPL - you can send a single request and then dig through the data - group it, slice it, dice it, sort it, etc., something seemingly simple like this would require a bit more effort with any other language. I have my editor connected to a Clojure REPL instance at all times; I never know when I might need it.
[...]
> It's... hard to explain how powerful this is, and how much better things could be if we had similar tooling in Python.
It would be kind of odd if we didn't, given what ecosystem Jupyter comes from. Sure, it's kind of surprising that the editor to integrations to do this haven't been around for a long time, but...
https://code.visualstudio.com/docs/python/jupyter-support-py
Look, I've been using Python professional for 16 years, and ever since starting to use Jupyter 7 years ago, have been a huge fan and proponent of Jupyter. But the Jupter model is different from the Clojure "true REPL" model.
The main difference is that in Jupyter, you're doing something that is outside the normal files you use to write code. Both by writing in cells, but also you're physically in notebook files which are not part of a "regular codebase". Sure, if you're doing standalone analysis work or something, that makes a lot of sense. But if you're, say, writing the BE of a webapp, then you'd have your normal Django or whatever .py files, and separate Jupyter files for one-off analysis, or for writing testing code, etc.
In Clojure, those can be unified in a way, both in the development process and in the future. Meaning, you'd start off with the equivalent of your Django .py file, you'd make a special "smart comment" which doesn't get executed normally, but which you can manually execute via an IDE command, you'd write some code there, and once it's working, you'd move it outside the comment to the "regular" part of the file. You've now essentially iteratively developed a new function that is just part of your normal codebase.
And the "comment" that you wrote all the testing code in? That comment is still part of your file. You can use it to document the way your code runs, but in an interactive manner - anybody chancing on this file can just execute commands out of that comment and immediately see what they do.
It's... hard to explain, but it's a really powerful model, that completely obliterates the line between "exploratory" code like writing in Jupyter, and the "production" code that ends up in your actual codebase.
If you've never seen it, I highly recommend you look for a good video example of Clojure REPL-driven development.
That's not the case here (it works in normal .py files.)
> In Clojure, those can be unified in a way, both in the development process and in the future. Meaning, you'd start off with the equivalent of your Django .py file, you'd make a special "smart comment" which doesn't get executed normally, but which you can manually execute via an IDE command, you'd write some code there, and once it's working, you'd move it outside the comment to the "regular" part of the file. You've now essentially iteratively developed a new function that is just part of your normal codebase.
Yes, that exactly will work in the VSCode Interactive Python environment and is what distinguishes from normal Jupyter notebooks even though it relies on Jupyter under the hood (=the code can't actually be in a comment if you want to use the integration to send it to the interactive environment, because Python only has single line comments so sending it to the interactive environment would literally send a comment, but it can be in guarded-as-unreachable code -- it could also be in a triple-quoted string, which can work like a comment, but then you'll lose syntax highlighting, completion, etc., while writing it.)
One thing I'm skeptical about is whether the way Clojure itself is written, with lisp-style S-expressions that are identical in look, is an inherent advantage for this kind of work over the more standard way Python works. I'm curious to find out.
But that isn't very powerful compared to a problem domain expert working in a language they like. If someone doesn't like lisp then they shouldn't use it and they'll probably do well.
I understand the value of mainstream languages at the group level (ability to distribute work through simple layers), but my brain doesn't like this way of working. I used to churn a lot on my own and I miss some of that (the speed and the ability to find creative / novel ways to solve problems).
Also, the biggest complaint by many; I have not found any cases where packages we were missing in lisp but are there in nupkg or npm actually sped up any work in the long run. Our work is very regulated (so we check, read and fork the stuff we use to make sure nothing bad lands in prod) and our integrations almost never have integration packages available and if they have them, they are out of date or very badly done because everyone just does it from scratch to make sure.
Maybe we are weird and that is fine, but, if there is one thing; we iterate faster than our competitors.
You mean c# debugging and stability are bad relative to your own debugging workflow (I can imagine how CL makes things way faster) ?
> lisp (and prolog by the way)
Talk about low odds, I don't know how many shops use both but that seems like a rare gem. btw you use swipl as your prolog implem ?
Do you do conferences discussing some of your work ? I'd be very interesting in following that :)
The other advantages are less true because dynamic languages are prolific and powerful. But also you never really grok JavaScript or Python until you study Lisp. It’s common to see what’s essentially C or Java written in python syntax.
And let’s be fair nobody in the 90s understood C++ including pg.
Common Lisp also lets you easily compile your language to assembly. To get that additional llvm benefit is at least 2 orders of magnitude of effort, but it’s probably not 2 orders of magnitude faster.
Some benefits of Lisp that are less commonly reported:
Lisp let's you imagine a way of doing something, and then just implement that with nothing getting in your way. You can go from something half-assed to a whole-ass in no time. The type system lets you develop quickly and optimize when it's time. You can run interpreted, compiled, and even optimize the rendered machine code.
Image-based computing let's you skip the database for a lot of things as you can just read data into an image at startup and have, essentially, a fast in-memory DB.
Language-oriented programming allows you to extend Lisp with nouns and verbs related to your problem domain. So you can express the solution in terms domain experts already understand.
The hugely underappreciated condition system is far more than a way to do error handling, but can respond to events and do flow control.
Modernizing Common Lisp is something everyone does. We each curate our own distribution from Quicklisp and develop in that. More coordination there would be welcome, but we're not writing in your dad's Lisp.
A kitchen-sink, multi-paradigm language is ideal compared to a more opinionated system in which there are official paved paths to do things. You get to spend more time figuring out what needs to be done than figuring out the official way to do it. If there is a downside to that, it's that you have to take apart enough flashlights to figure out how things work and can't just NPM somebody's afternoon project over and hope it works.
When you have a perfect spec and you just need to implement it efficiently, think C. When you're not totally sure what you're doing, want to explore the problem space and not paint yourself into a corner, think Lisp. Actually, think Lisp anyway because specs are always wrong, so you're always exploring. Lisp makes for living systems that grow and change organically. C is for making complicated but dead automata.
That's something that stuck with me from the SICP lectures: problem solving via wishful thinking. That's part of why I love Lisp.
The way I think of it, CL is the rare language that I don't feel is fighting me every step of the way.
Unless you already have a rare Lisp job, or you're retired, spend your time on things more employable.
> Perusing the various search results on 'CL vs Scheme / Racket' had me personally more convinced of CL for heavy-duty "real-world",
CL has more wins on this, but Scheme has, on at least one occasion, been used for large, important, rock-solid stuff, and super-productively. It comes down to the individual programmer(s).
> all the other languages have kept picking up nearly everything once-cutting-edge about Lisp
Much of the relative value of Lisps nowadays is the communities. They're generally made up of people who would be doing this even if software weren't a well-paying job. And there's relatively little BS to wade through (like explodes in all directions around popular employable platforms), since most BS follows the money.
I think you hit the nail here. People credit lisp because it is, or was, the language of the super high end geeky dev. But it's the high end dev self-selecting into lisp, not the language itself, that is the real advantage IMO.
Find a really good developer and let them use whatever they like. I've known the proverbial 10x (or even more!) dev in a number of languages, even "blub" ones like C#. A motivated, highly competent developer with an eye on the vision and skin in the game is the real "secret weapon" no matter which tool she happens to choose.
Unfortunately, it's rare that a company can identify a really good developer.
Also, trusting a developer to choose the programming language for a project might not be a good idea right now, due to a pervasive industry culture of resume-driven-development and job-hopping. (Unless "really good developer" implies conscientious professionalism and alignment with business needs, and you're able to discern that.)
> A motivated, highly competent developer with an eye on the vision and skin in the game is the real "secret weapon"
Agreed. Though founders tend to think they deserve 20x+ the equity of even first hire engineers. To the point that even your typical early hire is probably best off freshening up their resume keywords (via resume-driven-development), and job-hopping as soon as a better compensation package comes along (maybe after vesting cliff). The founder already signaled that they think the developer is a commodity, not a secret weapon of the company. That's not motivation for the hire to be aligned with company success, much less beyond.
1: https://en.wikipedia.org/wiki/OODA_loop To a certain extent this is tooling, but CL, at least (I can't speak for other members of the Lisp family) was designed with a peculiar combination of dynamism and efficiency in mind that nobody else does. It's in many ways more dynamic than Python and yet easier to write an efficient compiler for, which is quite the feat.
I've never seen lisp in production, but if I had I'm sure I'd be like, ok, we use lisp here.
Lisp is just a tool, and some people with find it ergonomic and powerful, and others will find it awkward and cumbersome. Use the tools that make you happy and productive and don't worry too much about trying to find the best in an ambiguous space.
Nothing is objectively cool, it's super great that you enjoy it. It's also fine that other people don't.
Okay, let's try to go through each of them more or less objectively, shall we?
- "Lisp is archaic and outdated"
While Lisp was introduced in the late 1950s, it has evolved considerably. Modern dialects like Common Lisp and Clojure offer extensive libraries and features that compete with contemporary languages. New Lisp dialects are constantly being added to the family. Clojure-Dart and Jank are more recent examples.
- "Lisp is hard to read"
Lisp’s syntax, based on simple and consistent S-expressions, can be highly readable once you familiarize yourself with it. The concise and uniform structure allows for straightforward parsing and manipulation of code. Try taking any JSON and convert it to EDN, or any HTML/JSX to Hiccup, and visually compare it - the latter would look far more readable. One doesn't even need to know any Clojure at all, to see that this is true.
- "Lisp doesn't have a powerful type system"
While Lisp is generally dynamically typed, Common Lisp supports powerful optional type declarations and checks. Dialects like Typed Racket offer strong, static typing. There's a new extension for CL called Coalton, and Clojure has Spec and Malli. You can do mind-blowingly awesome things with them, e.g., building a data structure for a ledger where each transaction is ensured to be in the correct order and shape — something that would require significantly higher effort to build in any other programming language.
- "Lisp code is not easy to refactor"
The homoiconic nature of Lisp, where code and data share the same structure, makes it particularly amenable to metaprogramming and automated refactoring. Tools like SLIME for CL and Cursive for Clojure can do an amazingly good job helping you refactor.
- "Nobody uses Lisp in production"
While not as mainstream as some other languages, Lisp is actively used in various domains. Companies like Grammarly use Common Lisp, and Clojure is widely adopted in the industry, particularly for data processing and back-end services. Apple built their payment system, Walmart their billing, and Cisco their entire cybersecurity platform. There are tons of examples like that.
There are many seemingly great ideas in our industry that initially received hype but later faced significant criticism — goto statements, the waterfall model, object-oriented databases, CORBA, SOAP, XML everywhere, and so on. Lisp, though, despite all the skepticism, keeps coming back again and again, and just refuses to die.
So, I stand by my words: Lisp (as an idea) is really awesome. Every programmer should gain at least some basic understanding of what it offers, because there's no harm in it — only benefits. Again, I'm not fighting anyone disliking any concrete tool, but hating the idea? Why? The only explanation I have for that - shallow understanding of what that idea is about.
Indeed, there is no language that is just "Lisp" any more. There once was, of course.
There is a legend that he requested that no new language be called Lisp (and nothing else) so that there is no confusion.
SBCL, Racket, Gauche Scheme, TXR ... are examples of concrete tools for specific tasks.
Without knowing what (if anything!) they don't find awkward and cumbersome, that is not very useful.
Data is very biased in this space, because the Lisp family receives a lot of drive by visits from language nomads. A good chunk of those people find programming awkward and cumbersome. They are having a hard time, and blame the tools.
Lisp is not going to fix someone who can't learn C++ or Java (or whatever) to save their life, but blogging about their disappointing experience will generate flak for Lisp.
I'm saying that if you don't find it easy to edit Lisp, you can't be "normal". You must have dyslexia, ADHD or whatever. Show me how well you can work with C, Rust, Makefiles, Shell scripts, CSS, JavaScript, Kotlin, assembly language, and whatever else before I believe that you're having a problem with Lisp and nothing else.
Lisp is not a refuge for those who can't hack it. I can see how some people might make that false interpretation, by a false association of tenuously related ideas. A language or language family that is under-used is not automatically something that is suitable for or used by the inept or the misfits. Programming language fringe and cognitive fringe are totally different categories.
At the same time, the state of the art in programming languages has advanced since PG wrote those essays. Even many of the languages which he was presumably referring to as "Blub" are much better now than they were.
A bigger issue is: As one gets better and better at computer programming, at some point the specific implementation language starts to matter less and less. (It never completely ceases to matter; but, less and less.) In one of his essays, PG contrasts "applying Bayesian filtering" with "applying if statements", which describes the matter quite aptly. At some point, language constructs such as "if statements" really do melt away.
For this reason, I reject claims that using a certain programming language will make one "1000 times more productive" (or similar) out of hand. The big problems in software are not at the level of the language at all.
This may sound false to newer programmers who have just discovered a fantastic programming language which allows them to work much faster than they could before. To such ones: I've been in your shoes. I've tried many languages, perhaps even the one you are enamoured with now. The benefits don't scale indefinitely. At some point, as you apply your fantastic new language to bigger problems, you will find yourself wrestling with the same issues as you used to with your old language, perhaps in a different form.
I enjoyed every single programming language I encountered and decided, or was compelled, to learn. Starting with BASIC, then Pascal, Assembler, C, VB, C#, F#, Java, Javascript, that followed by the plethora of "script" langs — Coffeescript, Typescript, Livescript, IcedCoffeescript, GorillaScript, etc. After that came Python and Haskell. Almost every book promised unbelievable riches and magic with this new (to me, at the time) language. Sooner or later, though, the long honeymoon would come to an end. At some point, while dealing with yet another clusterfuck of a project, I began to feel that I was simply not a good programmer and that I would never become one. Then I discovered Lisp. Somehow, inexplicably, things started making sense. Shit that confused me about Haskell and Prolog, I for whatever reason have started to grok, or at least I felt like it. For the things I build today, I use Clojure(script), Fennel, Elisp and CL. Even when I'm required to produce code in a different language, I still first prototype it in a Clojure. Not because I'm obsessed or blindly in love with one specific programming language, but because for the applications I'm building today, it makes sense — for me. I wouldn't argue that it would make sense for everyone, no. But for me, it does make sense. Today. Tomorrow that may change. Any given programming language is very unlikely to make you ten times more productive or accurate, yet it may provide you with even greater satisfaction. How? By allowing you to build something you love with it. Build something. Anything. And keep building more. And if you don't love it, rebuild it in a different language until you do.
IMHO this only partially true. Think e.g. of memory safety.
But nowadays almost every popular modern language is memory safe. So it has become yet another thing that doesn't really differentiate Lisp anymore.
The real time sinks are caused by the same thing regardless of language; poor design.
There are things that you can do in more advanced languages which are simply not possible in the simpler languages. In that sense, e.g. Java is lower-level language than Scala. Easier to learn and probably faster too, but just much less expressive. I've seen expert Scala dev doing some magic in 100-200 lines, which literally saved us thousands of lines of code that would be a nightmare to maintain and which would be a neccessary evil in a lower-level language.
Caveat I have never written Lisp professionally, tho I do use function generating functions when I can. E.g. an API with 20 methods that all do the same basic thing with little variations. I will write one big function generating function that takes parameters for all the variations and then just assign the dynamic output to the expected API name (in Python). I have a go test suite where it runs the AST over a directory and extracts all the functions with a certain name and generate a list of tests then all my components run the dame test suite but calling different implementations for service calls. Or test generators where you specify parameters like expected output and input and implementation details and get an array of callable tests in nodeJS. So my code isn't trivial for outsiders to read but easier than a very DSL macrofied lisp would be.
When you buy a plane ticket on Kayak or Orbitz, you're traveling thanks to Common Lisp.
fun fact: there is a Common Lisp NFS Server inside the Boeings 747 and 777.
https://github.com/azzamsa/awesome-lisp-companies/ (nothing official)
a new one: when people in Norway drive under tunnels, they do it safely thanks to Common Lisp.
I personally don't understand why anyone who has had the first-hand experience of writing software this way would willingly give up on this fun. Yet, I also understand that personal impressions can be deceiving. Just because I love chocolate and every single person I have ever met also loves it, it doesn't mean that there aren't people who genuinely hate it.
Sure, tons of other programming languages have borrowed features from Lisps and continue to do so, but there exists an emergent category of programs that are assuredly more difficult to build using non-Lispy languages. Check out, for example, hyperfiddle/electric.
My biggest personal regret is that I discovered Lisp too late in my professional life. For a very long time I have had the same experience you're describing — starting projects with high promise but then hitting practical roadblocks. With the caveat that none of them were in Lisp languages.
Today, I don't even think about it — if I have to build something, I'll grab a Lisp dialect. I don't consider Lisps to be different languages; I can relatively easily switch between Clojure, Clojurescript, Clojure-Dart, Fennel, Common Lisp, Elisp, etc. I can't say the same thing even about Javascript and Typescript. Darn, even when I have to switch between different JS libraries, it feels like I have to deal with some mental overhead. That doesn't happen with Lisps.
At the end of the day, there are always trade-offs, whatever tools you choose and whichever language(s) you pick. Nothing is ever perfect. For me, if something isn't working, then maybe I need to find a Lisp better suited for the job. And usually, there are always options to choose from.
And I think the core thing that makes it possible is that the design decisions in lisp, optimize to provide expressive power to the user, because as a language designer I cannot perfectly plan what you will build.
Which in turn enables the users to build things that would require crazy effort in more traditional languages.
And yet, most programmers have an irrational fear of built-in metaprogramming because it "makes hard to understand code". But seemingly nobody is afraid of bad metaprogramming, like code generation. Which I would argue is more difficult to understand.
Seemingly, any sufficiently hard problem requires some new language constructs to be built to discuss it in a way that makes sense. A good example of this is react. It's a framework in JS, but when using react you aren't really writing normal js. You are writing react. This is magnified when using jsx, which few people will complain about using, as it makes the process of writing react code much more enjoyable.
The difference here is that “bad meta programming” happens when the people writing thorny libraries and frameworks clamor enough for meta programming that the language implementers shoehorn in a subpar implementation. These people know how important it is to programmatically generate and modify code, because they had problems that just couldn’t be solved without that, and the other language users never see it (very few programmers in most non-Lisp ecosystems are curious about the code of libraries they use) and so don’t care whether or not it exists.
On the other hand, a language with pervasive meta programming is going to have cases where you could do a task without that feature, but can do that task much more easily with it. Meaning that of the programmers using that language, everyone in the subset of “programmers that like to minimize what they need to learn as much as possible” will be exposed to code using meta programming.
Most of these engineers will not have the prior experiences to understand that “ok, we need to learn meta programming either way because it’s essential to many important coding tasks and so has to be in the language; it’s not an increase in learning/comprehension difficulty to also have it in code that doesn’t strictly require it”. As such, their reaction is simply “why do I have to learn how to work around this extra thing?! Get rid of it, now!!!”.
REPL-based development in a language that does not have large scopes of nested things is just so great. You can redefine everything on the fly. That is incomparable to restarting whole programs. Scala also has a repl but it not nearly as useful with idiomatic Scala code, because lot's of stuff is defined in nested scopes and by inheriting and extending trait. After a simple code change you have to recreate lot's of state. That does not happen in Clojure when you work primarily with builtin data types and lots of functions. This advantage is not only based on the REPL, but also on the way the syntax of is designed.
I work on massive C# website that takes forever to warm up and get running. Stuck on old .net framework. A rebuild/rerun of the website after each little change is incredibly frustrating and slows down the feedback loop. Fortunately Microsoft implemented hot reload, and it has been a game changer. It still has all sorts of quirks and gotchas, but despite that I love it.
If there is one thing to take from common lisp, it's the live image development style. It's unfortunately hard to bolt on to languages not designed for it from the ground up.
But I don't know your situation, just sharing that .NET Framework based products are dreadful to work with in 2024 over .NET 6/8.
Perhaps your imagination would benefit from some elevation. Try shrooms or something :) I think your understanding of what a REPL actually is might be somewhat skewed due to a shallow understanding of how it works with Lisp dialects. The experience of working with the REPL in Python, for example, is not equal to the experience in e.g., Clojure.
There are many different companies that leverage Lisp dialects to build their businesses. Companies like Cisco, Apple, Walmart, Nubank, Grammarly, Funding Circle, Pitch.com, Rate.com, and many others are not using it for "quick-and-dirty one-off stuff"
F# actually has this and while it probably is not as powerful as Slime with CL it can give an idea of the power you are missing (and F#'s has gotten some nice improvements in the script side of .fsx files such as being able to add nuget files with a line in the script in newer versions).
Yes, and it's a lot more than just typing in one liners (as the person below suggests). For instance, in standard C you can't redefine functions/procedures at runtime (e.g., to add debugging code, or even switch to an updated version in a running system without requiring a restart). Some of the more modern languages have that facility, but it's rarely as elegant as it is in Lisps. It's a consequence of functions/procedures being first class values, and is really hard to do (or do well) in any language where they aren't.
Can you explain how it is a consequence?
Note that I did say "Some of the more modern languages have that facility, but it's rarely as elegant as it is in Lisps."
Example: JavaScript is sometimes claimed to have first-class functions. It kinda does, but you can't redefine (e.g.) the "+" function as you can in Lisps. So it's basically half-assed. C has function pointers, which again is (sort of) like first-class functions, but also don't let you redefine "+" (they're also incredibly ugly to use). C++ has operator overloading, so you can (again, sort of) redefine "+", but unless things have changed since the last time I used it (which has admittedly been a long time, though not long enough), I believe that's only available at compile time, not run time.
And of course even in (e.g.) JavaScript when you redefine a function it doesn't automatically get compiled, as it does in many Lisps (maybe the JIT will eventually get around to it, maybe it won't).
And so on.
(I'm using "function" here as a shorthand for "function or procedure" because I don't want to type that a half-dozen times).
Lisp doesn’t even have "operators" — everything is a function. Gosh, the more you learn about Lisp, the more it feels like we went backwards. We tried to simplify programming, only to find ourselves buried under layers of complications. Just think about how many times we've reinvented state management for React alone: Redux, MobX, Recoil, Zustand, Jotai, Effector, XState, Overmind, Hookstate, Easy Peasy, Rematch. And that's not even a comprehensive list. What's funny is that some of these are "improved" versions of previous "improvements". Each time we try "improving" the previous "improvement", it seems we continually need to reinvent another layer of "improvements". Dafuk are we doing?
Don't learn wrong things. Lisp has operators and no, not everything is a function. LET for example is not a function. It is a built-in special operator in Common Lisp. Similar in other Lisp dialects. Scheme R7RS documents "let" as syntax: https://standards.scheme.org/corrected-r7rs/r7rs-Z-H-6.html#... with a "named let" as a variant: https://standards.scheme.org/corrected-r7rs/r7rs-Z-H-6.html#...
In Emacs Lisp: https://www.gnu.org/software/emacs/manual/html_node/eintr/le...
"The let expression is a special form in Lisp..."
Also: limiting the redefinition of standard operators is not backwards, it's useful, to prevent to crash a Lisp, since in Lisp function names are often late bound. -> changing a core function will have immediate effect on all code using that.
Macros are also not "function". Macros are operators which expand source code to new source code.
Lisp typically has these operator types:
Additionally one my see fexprs, which are operators, which are called with source code. Then in some Lisps some functions lazy evaluate their arguments.The glossary of Common Lisp explains "operator": https://www.lispworks.com/documentation/HyperSpec/Body/26_gl...
> You can't redefine itIn standard Common Lisp it is UNDEFINED what happens when one redefines a standard operator of Common Lisp. Typical implementations will signal an error (which often provides a restart). Typical implementations will also extend this to other protected packages, such that the user can't accidentally replace an operator and the Lisp system may crash.
When a is a class type (and more recently this is now supported for enumerations also) then the surface syntax a + b means the same thing as a.operator+(b): invoking the operator + member function of a with argument b.
Unless I'm mistaken (which I may well be), you can't do that at runtime, only at compile time.
Indeed, in C++, our a + b expression is statically analyzed to be working with specific types; the declared types of a and b. When that code is compiled, it will work with no other types.
However, if we anticipate that kind of extension, we can have a be a reference to some numeric_base class where operator + is a virtual function.
With some help from a bit of platform specific coding, we can have dynamically loaded modules which can provide new classes derived from numeric_base that ca be passed to existing code in the program which uses + on numeric_base.
I don't think lisps strengths can be matched by piecemeal feature adoption. The GC, the REPL, the ability to reload code in a running process, the macro system, the minimal syntax - it _all_ comes together and it shapes your development process.
Case Study: My finest work was written in Clojure, I'm pretty sure I wouldn't have a clue how to achieve the same in Go, Rust or whatever else the broad programmer base is typically interested in.
The single most challenging project I've ever worked in, domain-wise, had so much churn in the spec because we laid the track as we went. We changed data storage products (Postgres -> Datomic), we frequently re-wrote the UI (back when ReactJS was popular and Redux etc were being proposed). We re-architected mid-way to use event sourcing as we needed to be able to recreate the system state at historical points in time. And so on. A lot of dysfunction as well. But Clojure was probably the biggest reason we managed to limp along. I can't imagine having to r-earchitect so furiously while pleasing a borrow-checker which tends to make refactoring much more extensive.
Recommendation: If you want to give it an honest shake, but you're afraid (as I read it) of painting yourself into a corner, consider Clojure. Clojure is a fringe language, like all Lisps, but rides on top of two massive ecosystems: the JVM (Clojure) and Node (ClojureJS). Additionally, if Emacs/(Neo)vim is not your thing, there's an IntelliJ plugin, Cursive, and it all works _beautifully_.
Additionally, the community, and its designer, have many interesting takes on software design which I found very educational.
Where I think Lisp shines ?
* When things are _complex_ or projects drift, revise and change often. The ability to be functional, to build custom DSLs, border-line custom interpreters with way less effort pays off.
* If I had to teach programming to _complete_ newcomers, gun to my head
* When valuing _genuinely_ different perspectives. Picking up Lisp offers more than yet another C-like language
* When you have limited mental capacity for more stuff, i.e. you have a family life and a complex, demanding job already - maybe pick something that won't drag you through borrow-checker nonsense or category theory.
Observation, why programming in the large tend to suck: Why normal languages kind of suck. --
Essentially, enterprise code tends to require a lot of lines, those lines can have bugs. People want to reuse code (write less code, really), so they start writing abstractions, leveraging the many different (flawed) tools their language provides. These are typically a _lot_ harder than Lisps function composition and macros (say, Rust macros, Python metaclasses+inheritance), this means complexity goes through the roof. Over time requirements shift and abstractions become less suited, now you're fighting on two fronts. I would say you either need Lisps ability to write terse code (macros being the unique feature), or you need a _good_ code generation tool. And we seem to have neither in most projects.
Conversely: where is Lisp less useful ? --
The well-paved roads where a popular language has a set of finely developed libraries for solving your issues. But once your domain itself, your business logic, gets complex, there won't be a NPM, PyPI, or crates.io to save you.
I chose common lisp. It's very expressive which means my development velocity was very high with minimal boilerplate. I love being able to build functions up from the inside out in the REPL, and the DX of doing this in emacs is awesome because of the mature support built into the editor.
The library ecosystem in lisp is not stellar, and I think this is lisp's biggest shortcoming. With that said, there's still robust libraries for threading, networking, JSON fuckery, and (importantly for my use case) statistical analysis.
Lisp is great and the SBCL compiler produces fast code.
Writing in LISP can benefit from, I believe, a kind of psychological stance of wabi-sabi (non-perfectionism), for (barring exotic LISPs with sophisticated type systems), it will always be more challenging to refactor and get things just right than with languages like Rust.
Given the pressure of writing code for firms that can demand we are perfect or pay the price [being on call, etc.] (ie working for perfectionistic firms with perfectionistic customers — it’s a recursive problem) also can push us towards languages with powerful compilers.
Then again, maybe we need to invert things — use the languages we love that are not “perfect” but are so easy to love; for perhaps that love can just as recursively emanate outwards as would-be perfectionism…
Take for example a trivial example when we need to validate user data with the following constraints:
- Age: Must be between 18 and 120.
- Name: Must be a non-empty string.
- Email: Must follow a valid email format.
- Credit Card Number: Must pass the Luhn algorithm check.
Something like that is pretty straightforward to achieve in Clojure.spec within 20 lines of code. It's close to impossible to get with any advanced static type system, because type systems can't deal well with custom constraints; they often can't offer detailed, human-readable error messages for the cases like that; they often can't describe complex and nested data structures, or trying to do so would be cumbersome and less expressive.
So, they are not types. Okay. Cisco built their cybersecurity platform using Clojure. The market capitalization is about $180B. I imagine this would be a few dozen billions more if they had chosen to use an "actual type system", amiright?
----
edit: After thinking about it a bit more, I have to admit that I spoke too harshly. It's not "close to impossible" You probably can create a type that checks for these constraints, but you would likely need to use a dependent type system or refined types, perhaps something like LiquidHaskell., and it still would be very tricky, especially with the credit card part. In Clojure.spec - it's very trivial.
For example... one can easily express all that in Common Lisp:
https://www.lispworks.com/documentation/HyperSpec/Body/04_b....
also, avoiding SATISFIES: It then has assertions integrated with the condition system, which makes this easily checkable at runtime.SBCL has this extended with type declarations as assertions: http://sbcl.org/manual/index.html#Handling-of-Types
Eiffel had promoted Design by Contract for a long time.
> Cisco built their cybersecurity platform using Clojure.
Cisco uses a lot of software. Your Clojure marketing attempts are getting silly.
But then, the standard /is/ useless.
There are implementations which check these type constraints. So it is best to check the manual or ask the maintainers. IIRC there were implementations which added it on user requests.
But don't think all "Common Lisp" implementations will check those constraints, for example ABCL does not. Some don't and in others it can be turned off.
A few years ago I had a choice to learn lisp or smalltalk. It was a coin flip but I went for smalltalk
The downside is that it's a difficult tool to become fluid in and so it's difficult to find experienced hackers, so scaling the team is going to be slower than with other, more popular languages.
On the other hand, the ROI from hiring a couple of developers with Clojure/CL experience is almost always guaranteed to be higher than hiring a bunch of "regular" programmers. Niche languages like Clojure usually attract more experienced, tired, "seen some shit" developers who have likely worked with multiple languages before. Their broad experience typically leads to higher productivity and better-quality code.
There are alot of caveats here, and it highly depends on the teams, projects, maturity, quality of tools, and so on, but there are not alot of weird concepts in clojure that makes it that hard to understand (as a counterpoint to eg: rust borrow checking rules, or C++ templating, and so on).
The hard part with clojure is having just enough discipline to keep things in check.
No one ever got paid for using Emacs and writing Elisp (except for that single Kickstarter campaign for Magit). You can get paid to write in different languages while still hacking on Lisp dialects for your own satisfaction.
Need to configure a Lua-compatible window manager? Fennel is a great choice. Trying to scrape pages from a website? You can start a Clojurescript REPL and interactively identify elements on the page, "click" on them, "fill out the forms", and finally build a concurrent, async pipeline to sift through hundreds of pages and their content. Need to explore an API? Instead of dealing with JSON directly, you can use Babashka and, interactively, examine the results — slice, dice, group, sort, and filter through them in a way that would be extremely difficult with tools like awk, jq, and others.
Everyone may have their own distinct idea of what defines a "hacker's dream," and often developers gravitate towards tools they find comfortable, often falling into the "everything is a nail" trap. The idea of Lisp, however, with some practice, feels more like a factory for making your own hammers rather than a concrete hammer on its own.
The other thing that pg keeps hammering on about is Lisp Macros which I am not well versed enough to properly say how good of a feature they are. The alternatives most other languages have are much less powerful, but at the same time it seems that macros are very... abstract-y?
Modern software design thinking says that abstractions should be avoided when possible because they make the code harder to grasp for people not familiar with the abstraction on top of making the code too rigid and hard to modify once new requirements come in.
The alternatives to Lisp Macros like codegen, method annotations, Rust Macros. Are all much less powerful and usually harder to create, but maybe that is a good thing.
Wouldn't it depend on application domain what's a killer feature & what's not?
"Include if lightweight & very useful" seems appropiate. But what's considered "lightweight" varies depending on who you're asking / where you're looking, doesn't it?
Like passing a function to any function that iterates over a data structure (like array sorting), or registering an event handler. Sure you can usually achieve the same thing without first-class functions but it feels so much less concise.
I just love first-class functions so much that the main reason I dislike python is that it doesn't include proper anonymous functions.
It is definitely much harder to do it compared to Lisp-like macros though. But maybe the barrier to entry to create such powerful abstractions is a good thing. In my previous post I didn't say you should never use powerful abstractions, you should just not overuse them.
The main problem we were trying to solve is essentially that we had a lot of code like:
f(x) = sma(ewm(x))
g(x) = slope(ewm(x))
And we wanted to avoid recalculating ewm(x), while also not holding memory for it for longer than necessary. Our production use cases would have tens of thousands of intermediates, so the potential saving from getting this right is huge.
Naturally a DAG is a great fit for this type of problem. We started by creating an explicit DAG, where everything like ewm(x), sma(ewm(x)) would be a node in the DAG. We could then evaluate the DAG efficiently using multithreading, dropping nodes when everything that depended on them had been calculated, etc.
And I would usually stop there, but there were two main challenges. One was that the syntax was pretty clunky, and we wanted researchers (who weren't software engineers) to be able to use it. The other was that we wanted _any_ reference to something like ewm(x) to map to the node, even if they were used in very different places – and it was quite difficult to map these dependencies without explicitly tracking.
It's hard to quickly summarize our macro, but basically what it let us do was write the short syntax and track dependencies using the underlying Julia expressions. We mapped those expressions to nodes in the DAG, and that let us calculate the dependencies without having to track them explicitly. I don't think this would have been possible without some amount of homoiconicity and macros.
I kinda agree that macros can lead to a lot of (bad) abstractions if you overuse them, but when you need a macro a lot of times the alternatives a language provides can somehow feel more magical if you try to get them to do macro-y things (decorators/attributes... languages always seem to mix those two terms) or just cumbersome and in my opinion kinda an ugly hack to paper over the fact that your language is lacking in some way (codegen). I'd much rather read a simple (lisp) macro then understand how your ugly codegen system works.
I dont have enough knowledge to comment on rust macros though.