The thing is that trying to understand "the power of Lisp" via the toy interpreter from The Roots of Lisp[0] is like trying to understand "the beauty of the sea" after seeing a single five-minute YouTube video explaining it.
It will give you the basic idea, but it won't tell you about macros, reader macros, compiler macros, and having the whole language always available to build your abstractions on; it won't tell you about CLOS, including multiple dispatch, metaclasses, or method combinations; it won't tell you about the condition system and programming from inside the debugger; it won't tell you about live recompilation and live coding; it won't tell you about the JVM interoperability of Clojure, Kawa, or ABCL; it won't tell you about Racket's tower-of-languages approach to programming; I can keep on going for a while. And, since most Lisp dialects and implementations are nowadays compiled, either to native code or to bytecode, the just-interpreting approach doesn't even work very well.
Articles like this are fun, but they end up completely missing the point. This isn't "the power of Lisp"; it's just the very, very, very beginning. The rabbit hole goes much deeper.
Symbolics provided Pascal, Prolog, Fortran 77, C and I believe Ada (but I never seen that one) implementations that used ZetaLisp as intermediate language.
I've learned Clojure. Tried and failed to see this unique power of macros.
Truly asking for help: can you help explain what can I do with macros that I cannot do with functions? Or, maybe, cannot do with high quality or low complexity using functions?
My impression, and I hope I'm wrong, is that macros and meta programming were powerful ideas 30 years ago, but now most modern languages have generics, template meta programming [1] and reflection. I'm a curious amateur and probably out to lunch, but there you go.
[1]
Some other languages support similar, if not more powerful, compile-time facilities (such as Lisp macros), but those are outside the scope of this article.
The existence of template metaprogramming is an argument for macros, not against them. It shows the lengths to which programmers will abuse and misuse a feature so that they can reach their goals.
When I first started writing asm code, it was accepted to write self-modifying code (programmatically create new code, then jmp to it.) Then I discovered and was told it was a bad idea. So that's how I view Lisp macros - creating code on the fly and executing it.
When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it.
In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon.
> So that's how I view Lisp macros - creating code on the fly and executing it.
This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct.
[0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated.
[1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
Thank you for taking the time to answer! At compile time vs at runtime is an important distinction that I need to think about. I'll start the video from the beginning because I'm curious about Clojure and most things Rich Hickey does, since I liked is Simple Made Easy video so much.
So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along.
If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL.
Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears.
I'll keep learning. I'm sure the light bulb will go off eventually.
A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional.
Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros.
Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure:
(when false (infinite-loop))
If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop).
But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial.
(defmacro when
"Evaluates test. If logical true, evaluates body in an implicit do."
{:added "1.0"}
[test & body]
(list 'if test (cons 'do body)))
So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler.
So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op.
This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it.
Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does.
Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages.
(defmacro with-open
"bindings => [name init ...]
Evaluates body in a try expression with names bound to the values
of the inits, and a finally clause that calls (.close name) on each
name in reverse order."
{:added "1.0"}
[bindings & body]
(assert-args
(vector? bindings) "a vector for its binding"
(even? (count bindings)) "an even number of forms in binding vector")
(cond
(= (count bindings) 0) `(do ~@body)
(symbol? (bindings 0)) `(let ~(subvec bindings 0 2)
(try
(with-open ~(subvec bindings 2) ~@body)
(finally
(. ~(bindings 0) close))))
:else (throw (IllegalArgumentException.
"with-open only allows Symbol...
Thank you for the examples. I'm really curious to get to the bottom of this. My inspiration has been Erik Meijer, specifically his article "Confessions of a Used Programming Language Salesman. Getting the Masses Hooked on Haskell" [1]. If I understand you correctly, you are implementing lazy evaluation in your when macro, something you call short-circuiting.
By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it.
As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages.
I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try.
I'm happy to chat more if it's helpful (contact details in profile). I'll be honest that I didn't get Lisp for a long time, and stilll wouldn't claim true expertise. I attempted SICP a few times based on the glowing reviews from folks I admire. One of my paths to tech was via Paul Graham's writings, so I had a bias for Lisp early on.
Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros.
First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation
Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility.
Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp.
It's simultaneously very prosaic and very deep. Macros are pretty simple:
1. Access to the AST of a chunk of code for arbitrary processing.
2. At compile-time.
This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion.
Where Lisps are different:
1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures.
2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation.
3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally".
4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1]
These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build proces...
> homoiconicity... is simultaneously very prosaic and very deep
That's what drives me on. That and Rich Hickey's enthusiasm and the sense that he knows something really valuable that I don't. I watch a lot of videos as well, so I can't remember who said that programming languages were user interfaces between human and machine and are an attempt to talk to machines to get them to do things. In that sense, programming languages are a lot deeper than syntax convenience or efficiency. A good language allows us to talk to machine at a much higher level, and I get from functional programmers that this is what they find so attractive.
I will contact you if I get really stuck and when I finally get it just to share it with someone. Thanks!
edit: I've heard a lot about SICP but never looked into it. I think it's time, now that I have time
I don't have a computer science degree and I think that has held me back some, but I try to overcome it by always trying to learn and never thinking I should already know something or feeling like I'm an expert. I don't have the patience at my age to go back to school and get a degree. That you attempted SICP a few times is a good sign that you don't give up and speaks to your success in finally understanding.
Homoiconicity is neat, but I think the real nice thing is the S-expressions and general lack of special keywords. It makes it easy to take a chunk of code in your source file and seamlessly send it to the REPL and have it work.
Compared to Elixir (another language I like) - it is homoiconic but it doesn't have this feature. I can't, for example, take an arbitrary function definition and update or define it in the REPL - because all functions must be defined in a module. However, I like these modules because I think it helps enable discovery of functions - it's much easier to find all operations you can do on lists or maps in Elixir than Clojure.
Also, I wouldn't necessarily call macros more powerful than functions. They're just different, and they enable different things. You cannot, for example, pass a macro around like you can a function. Personally I've never written them, but I use them every day because the core library defines macros. I view them as ways to let people smarter than me to give me tools I can use to write my programs.
As far as driving you on, I don't think Clojure is the end-all be-all language. I happen to like it. I also like Elixir. I know people who really love other, more standard languages though. I don't think you need to force yourself to love or "get" Clojure, but playing around with it can expand your mind a bit, especially if you've never used an FP before.
But if you want an FP that (IMHO) would be easier to transition to, Erlang was the first FP that I "got", afterwhich I tried Clojure again. The syntax is a lot more like traditional imperative programs which can ease the transition. Nowadays I would recommend Elixir over Erlang for several reasons: more modern with modern tooling, less syntax quirks, more active online community, lots of active development.
For me, the main benefits of Clojure are (other FP languages have some of these too):
1. Sharable, immutable state as the default.
2. A set of higher order functions that can do all sorts of data transformation.
3. Being able to use the same language on the web in client + server (without having to use Javascript).
4. Clojure gives me the ability to, more than other languages I've used, target the level of abstraction that I need for the problem at hand, which removes a lot of cruft.
5. Lack of special syntax: helps enable REPL driven development, makes it easier to seamlessly adopt new features from other languages.
6. It's hosted on two ecosystems I'm very familiar with - Java and Javascript.
As for REPL driven development, it's not about typing code at a REPL. This video provides a pretty good example of working at a REPL in Clojure. Note that with the exception of one doc reference at the beginning, all code is typed directly into a file buffer. The editor provides shortcuts to send various forms to the REPL for evaluation.
Consider that macros generally don't expand syntax - because there is very little syntax anyway, and what you have is a tree of data and function calls.
A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need.
But they are still normal functions - it's the time of execution that changes.
I generally never used macros when I wrote C because the C macro language is lacking, there are a lot of pitfalls and saving function call overhead is usually not that important.
I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.
Writing macros in Lisp when you don't have to is also generally discouraged. They're typically harder to write and debug than functions, so their use is best reserved for situations when the opposite would be true.
Exactly this. New Lispers sometimes overuse macros because they're so powerful, but macro overuse can make your code impossible to debug or maintain.
At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated.
There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false.
*See pg's _On Lisp_ for an in-depth discussion of macros.
If you want to see some very productive uses of metaprogramming in Julia, check out Jump.jl or DataframesMeta.jl. Both use metaprogramming to create their own little DSLs that make expressing problems in a specific form very easy.
Self-modifying code isn't a bad idea. Like everything else, it has it's place, just not typically on modern computers due to breaking optimizations.
What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)
In my opinion, compile time codegen is way better than runtime codegen. Unless you are exploring genetic algorithms, runtime codegen is slow, brittle, and undebugable. Compile time codegen, however, can produce ridiculously fast code (see FFTW), and is much easier to debug because you can look at the generated code.
> Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming."
As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary.
Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.
Macros are not generics even though you can use them as such. Macros define new statements (and definitions for the languages that make a distinction). This allows you to extend the language to the specific domain and it's impossible to do with functions/generics in general case.
I can build a type system [0] with power approximately equal to that of Haskell within Lisp using macros. Not possible or even conceivable to do with functions.
From a Clojure perspective, here are a few things to think about and explore to help understand where the power and value of macros lie.
1. Clojure does not have all of its various conditionals as language built-ins. There is a single conditional construct built in to the language, if. The rest are all defined in terms of if. Function arguments are eagerly evaluated in Clojure. How do you write a short-circuiting function in Clojure? (when predicate consequent) as a function must evaluate both the predicate and the consequent. If consequent is slow to calculate then your whole when expression is as slow as consequent, even if predicate is false and we do not need consequent. when is a Clojure macro based on the if special form.
2. core.async is a mere library.[0] This implements CSP (same idea as Go's channels) and async programming (the sort of stuff that needs compiler support in other languages such as C# or Go) as a library. If Rich Hickey didn't write core.async, you could build this control flow yourself for async programming. In fact, you can create arbitrary control flow mechanisms using macros. Common Lisp's entire object system can be implemented in macros.
3. The only special forms in Clojure are def, if, fn, let, loop, recur, do, new, ., throw, try, set!, quote, and var. The rest of what you would consider the core of Clojure consists of functions and macros. Much of the control flow functionality is implemented in macros.[1]
The core of macros is that they allow you to evaluate arbitrary user code at compile time. Clojure code is represented as native Clojure data structures (the same is true for all Lisps). The entire standard library which you are accustomed to using to manipulate standard data structures is available at compile time to transform the data structures that represent your code. I can't explain this better than Rich Hickey does in [1], so I encourage you to watch the linked section of that talk.
[0]: https://youtu.be/yJxFPoxqzWE?t=560 See this video for an overview of core.async. Later on he talks about the stuff that is implemented as macros in the library. This link starts up when he's talking about C# style async.
[1]: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get an overview of what macros are used for in Clojure's core.
Maybe I am missing something. Here's how I might write something equivalent in Python:
for f in GetFromCache, GetFromDatabase, GetFromAPI, Compute:
try:
return f()
except Exception as ex:
log(ex)
return default_value
(You'd probably define a custom exception class for your application that represents the kinds of errors you can tolerate, so you don't end up swallowing programming errors, but you get the idea.)
Mmm. You've given a code snippet while Zak had a single macro call. Let's make your code into a function with signature:
def atdt_exception_suppression(*list_of_fns):
...
What you've done is leverage the fact that functions are first class objects in Python (ie, can be passed as arguments) to achieve the same result as a macro. Which is cool, it works. Probably slightly better than a macro for this use case because we get a neat function boundary.
But macros are a powerful option because it doesn't matter if functions are first class or not. If you are using a language where functions aren't first class objects, a macro system would let Zak implement anyway.
So you are correctly identifying that Python can implement this in an alternate way as a function, but a macro system is sufficient to do this and a whole batch of other tricks even if first class functions aren't available as a feature.
You’re running through a list of functions, the lisp example has a list of _function calls_ (which are not evaluated until necessary). Consider how your approach would have to change if each of those functions takes different parameters.
In ruby your lower level API should support methods with ! syntax that throw and without it that return nil instead:
def get_from_cache
get_from_cache!
rescue NoRecord
nil
end
Then you just do this to not only do it all but only do it once lazily on access:
def record
@record ||= get_from_cache || get_from_database || get_from_api || compute || default_value
end
Be slightly better if ruby supported a proper null coalescing operator like C# ?? and ??= or Perl // and //= but as long as you stay way from "false" as a valid value it works fine.
Assume, for purposes of the exercise that someone else wrote some of the methods you're calling and they do not behave this way. You could write your own wrappers, but that's a lot more code to write and maintain than a simple control structure that doesn't care what's inside it.
With macros, I can write a short macro regardless of how the upstream code behaves. Without macros, I can request a change to the upstream code and hope the maintainer will agree to make it behave how I want.
You can get macros out of rust or julia or other languages other than LISP as well.
It is also arguable that overuse of macros leads to less readable code since you wind up inventing your own personal language the more you do it, which requires more of a learning curve to work on it.
There's a large advantage to sticking with verbose idioms that are common and built out of a relatively few building blocks rather than building a meta-language which is terse.
At the same time I'm not saying don't have macros, but they should really be reserved for more extensive heavy lifting, not a pile of tiny ones used all over the codebase to make it terse.
A well written codebase should read more like Hemmingway at an 8th grade reading level using simple constructs from the base language which are put together, generally using programming design idioms which are common and terminology which everyone shares. There shouldn't be a lot of "what does this macro do and where it is it defined?" questions on every other line.
That would be the time, where I would consider to write a macro. The macro would capture similar logic, but would provide me no-overhead syntax.
In Common Lisp:
(defmacro return-first-working (&body body)
"Expects a sequence of expressions as body.
Returns the value of the first expression of body
which returns without error."
(when body
`(handler-case ,(first body)
(error ()
(return-first-working ,@(rest body))))))
(return-first-working
(error "hi")
(error "there")
(error "!")
"hello-world"
(error "wtf?"))
That way I can easily implement primitive control structures, without resorting to add to my code visibly complex things (like making expressions into functions).
Above is a recursive macro, which takes its subexpressions and returns a simplified expression, reusing itself. That way code transformation itself is described as a recursive process. One can learn how to apply such transformation patterns to the code itself and then it gets relatively easy to write.
I'm the one who missed something: each of my examples called its functions with no arguments. Your approach would, of course work when the functions are called with no arguments or the same arguments.
Each expression can be arbitrary code where to do something similar in Python you'd need to compute a thunk in advance.
There's also an advantage in clarity because an abstraction can be defined. Once you know what first-working is for, the purpose of this block of code is clear as soon as you read the first token. Using a pattern instead of an abstraction, it's necessary to read the whole loop to make sure it's really the pattern you think it is.
Spend 5 minutes playing with -> [0]. Appreciate how pretty it makes the code and how nice it is for laying out sequential operations. What a fun operator. Then try to implement it yourself as a function.
When you discover that you can't, you will have discovered the unique power of macros (note the docs link to source code so you can see the readable 8-line macro implementation). You've also learned a very practical skill, which is how to write threading macros that handle your special edge case fail conditions. It comes in handy from time to time.
Note that this cannot be written as a function only because Clojure uses strict (call-by-value) evaluation everywhere and has no way to change this (other than Macros). In theory languages could provide annotations to allow a user to specify whether parameters are call-by-value or call-by-need. Haskell and Scala both provide this.
But I don't think this is a bad example at all, I tend to think of macros as "extending the language/compiler", since they have access to the abstract syntax tree of their inputs, which is what is being done here.
Yeah. And for the interested reader, this is unique power of macros.
It doesn't matter what specific combination of language features are supported. Macros mean you can implement things anyway. It doesn't really matter what the evaluation logic of the language is, a macro can create syntax that does what you want it to.
If there is a macro system available, a new operator can be implemented in any language. If there is no macro system, it depends what features the language supports.
Colloquially: You can rewrite source code into a different form as it's being evaluated, then have it be further evaluated as if it were originally written in the new form.
One fairly simple example is Clojure's `when`, which is an expression that starts off looking like
(when (= 0 1) (+ 1 2))
and gets re-written to
(if (= 0 1) (do (+ 1 2)))
before being further compiled. This rewriting happens as many times as necessary until an expression "bottoms out" and can be evaluated in terms of only built-ins.
It's instructive to see how much of e.g., Clojure is implemented as macros. https://github.com/clojure/clojure/blob/master/src/clj/cloju... shows pretty clearly the correspondence between the original form and the rewritten form in the definition of `when`. `defn`, which is so foundational to idiomatic Clojure that it's easy to mistake it for a built-in, is a much more complicated example.
Note that functions in other languages, like Haskell and Scala, do allow some control over evaluation order. One can annotate function parameters to achieve call-by-value or call-by-need as required.
Macros can analyse/manipulate the abstract syntax tree of their inputs, so I think of them as extending the compiler.
Lack of macros makes nothing impossible, but also makes many things harder. I think of them as code generators, tiny compilers on a micro scale; they are invoked by the proper compiler during compilation phase, as opposed to runtime, allowing you to inject code that you'd otherwise need to inline by hand.
It's possible to write pretty complex things like control flow abstractions using just structures and functions/methods, e.g. a CL-esque condition system in Java which only uses classes, static methods, and Java lambdas[0] for its syntax. Possible, but also IMO ugly when compared to the CL counterpart, because the low-level but irrelevant details (such as instantiation via `new` or generics) are still presented to the programmer.
Author here, that's an apt analogy. Perhaps it helps to understand where I was coming from; I had never written a line of LISP and was struggling to fully grasp the power that Paul Graham had hinted at. So I started at the beginning, and although it was just the brim of the rabbit hole it certainly opened my eyes. And I hope it can help other beginners open their eyes as well.
I plan to fall down the LISP rabbit hole one day, and continue this series in full detail.
While I don’t disagree with Josh’s blog, concentrating on just language features leaves out the style of Lisp development: bottom up REPL style development. When I have to use Haskell or Python, I find myself working as if I were using Lisp: I still favor the REPL and building up from primitive functions to the top level application. This is probably a bad habit but it is the way I work. BTW, using the standard Emacs support for Python works very well: load a file, then re-evaluate just changed classes and functions (not Lisp, but a good work flow anyway).
I like Haskell, but my Haskell skills are weak. I experiment in the REPL to see what works and then write functions in an editor. Also, I like writing pure Haskell code, and when I need to do network or file access, etc., then I find appropriate code on the web that I can learn from. So, my use of Haskell is fairly simplistic. I even wrote a very short book using the small subset of Haskell that I am comfortable with and use for my own projects.
When I worked in Haskell I always found a type-driven top-down approach helped me work out what I needed to do literally starting from something like App :: Input -> Output
Out of interest where are you working such that you're using Lisp? Pretty sure I've actually asked you before ha, but can't remember. Every-time I get burned out with frameworks and polygot development I end up daydreaming of using a stable language like common lisp for the remaining ~40 years of my career.
I have used Common Lisp for a ton of paid work since 1982. That said, in the last 7 years people have paid me to do deep learning work, so that work uses Python. Follow the money.
I am deeply bound up in Common Lisp because of my history. However, if I may make a suggestion: you might want to take a careful look at Racket: easy to make standalone apps, good libraries, large community, and if you don't like Emacs, the Racket IDE is really pretty good. Sit in on one of the online Racket conferences to see many kinds of cool applications and libraries people use Racket for.
I've done a lot of Haskell, and my approach tends to be a mix of the two. There's a natural split:
• types let me play with the architecture and interfaces top-down
• the REPL lets me write more complex logic bottom-up
Usually there are a few "hotspots" in the codebase that require a lot of logic and are easier to develop bottom-up; the rest of the codebase is focused more on organization, domain modeling and information flow, where top-down design works really well. In actual practice I expect my mode tends to vary even more than that, but this division seems like a good approximation of how I think about it.
I feel like the bottom up style is possible in static languages like C++ too, except instead of manually testing small functions in the repl, you can write small unit tests for them instead. In a way it's even better because a test written once can be run many times, while testing in the repl requires a manual test with every change.
That works in Lisp, too though, but it's easier and faster to run the tests in the REPL, so you get the best of both worlds.
When I write Lisp I usually start testing in the REPL, iterate until the test code gets to be a few lines long, and then copy it over to the package's unit tests. Then I can keep iterating on both the test code and the code being tested.
"mylibrary.test" will depend on "mylibrary", so quickload will load both packages when they've been changed, and I can re-run the test quickly using Alt-P to call up the previous REPL input.
And the REPL is also really great for "what if" testing that you may not want to keep forever, like, "What happens if I call this function with an invalid parameter?" Hopefully there's a test for that, but it's often a lot faster to just call the function and see what happens rather than go dig through the test code and lose your focus.
constexpr and static_assert make this a ton easier in C++. If your "unit tests" are static_asserts, the IDE will just highlight the broken parts.
C++17 made this minimally useful, and things got spectacularly better with C++20 and constexpr vector/string. 23 is shaping up to move even more of the standard library to constexpr.
I wonder if we'll end up with something like https://www.circle-lang.org/ which lets you use pretty much the entire C++ runtime language at compile time. It certainly makes heavily templated code way more readable.
I'm guessing the draw is in the immediate feedback. Like iteratively creating a chain of commands in shell pipe. Compiling and then executing typically takes a minimum of a few seconds before you get feedback
I must admit although I was never much focused on either Lisp or Forth I tend to do lots of bottom up, no matter in what language. If I know I'll need a certain thing, for example a type to represent some kind of node or whatever, I'll build that thing before I write any code that would need to have that thing available in order to work. By the time I get started on the main task I usually have most or all of the necessary components verified. And I don't see why one should be apologetic about preferring that style.
I've been a serious Lisp programmer (recreationally and sometimes professionally) for around 10 years and I have programmed in Common Lisp, Scheme, my own weird dialects, Emacs Lisp, etc.
At this point I feel like almost everything written about Lisp is silly. Taken as a language family as a whole, there isn't much that separates Lisp from most of the other languages that are out there _except_ syntax-transformations or macros which, frankly, most people should never use.
For me, the best thing about Lisp is the regularity of the syntax. What I like least about almost all other programming languages is all the useless syntactic doodads and wingdings. The brackets, the indentation sensitive stuff, the unneeded complexity that languages insist on associating with what is and must be the denotation of a tree, in the end.
The other great thing about the language is that you must use `let` or `lambda` to introduce variable bindings and their scope is, therefore, always perfectly clear.
Despite these niceties, Lisp isn't going to make you into a super programmer. It won't really help you solve hard problems. It might help you quickly solve easy problems, but in the end, most of the job of a software engineer is grappling with conceptual problems with which Lisp is only sort of helpful.
As long as I'm holding forth, I will say that I enjoy programming in Scheme _much_ more than I enjoy Common Lisp, which feels incredibly, uselessly, and old-fashionedly complex with all sorts of weird corners to stub your toes on.
"Most people should never use [macros]" is not a very good take-away. This is silly and overly paternalistic, like most programmers are children who shouldn't enjoy an extraordinarily expressive programming tool because it's "dangerous to readability". I feel as though people imagine more macro horror stories than they've actually ever experienced. I've experienced just one horror story in an optional-use open-source numerical library. And so I just didn't use it because I couldn't figure out how to extend it for my needs.
Many macros—just like programs as a whole—are bad and unhelpful. But this has little to do with macros, and more to do with programmer technique. It's not intrinsically difficult to write a useful macro, but it does presuppose that the programmer has a useful syntactic idea.
The Lisp family of languages so so broad and loose. It's hard to say anything generally about them anymore. But if you narrow down to Common Lisp, there's still a treasure trove of unique features. Common Lisp and Clojure seem to be the only two lisps where you've achieved broad community agreement on tooling, ecosystem, and general library design. They're also two languages where you can get paid to write them. Can't say the same about Scheme.
Lastly, as a personal anecdote, I don't think I've stubbed my toe on anything in Common Lisp. It's a very safe language with a lot of language features. I have stubbed my toe with CALL/CC though and memory leaks.
What I mean about macros is that there just aren't a lot of problems that are really well suited to solution via meta-programming. People should be free to program however they want, but if we're evaluating the power of a language I think its fair to talk about the frequency and type of problems you might need to solve.
I think you're right about CL and CLojure, though. While I really prefer Scheme, its not really appropriate for solving a large set of practical problems.
I have to take issue with this. There are only two languages I consider when faced with a very hard problem: Common Lisp and Haskell. One can solve easy problems in any language. But when I have to solve a new problem from scratch that nobody's ever solved before, I don't reach for Java or Python. I reach for the languages that augment my brain rather than limiting it.
I agree. I have several times solved hard problems in something like Lisp or Haskell, only to have to port the solution to a more common language for integration later. While I would expect the port to take less time than figuring out the solution in the first place, it generally takes longer because the target language is riddled with gotchas and inefficiencies that make the actual programming more difficult.
Note that the examples in the "cdr" section are incorrect.
(cdr '(x a))
; does not return a, it returns (a)
(cdr '((x a) y))
; does not return y, it returns (y)
(cdr '((x a) (y b)))
; does not return (y b), it returns ((y b))
The author might be new to Lisp, and I hope what I write doesn't discourage him. There are some other mistakes:
((ab . c) . d . nil)
isn't a valid s-expression. Maybe it should be
((ab . c) d . nil)
Also,
(eq '(a b) '(a b))
; (a b) is a list and cannot be evaluated by eq
eq works fine on lists. It returns T iff its arguments are the same object (i.e. are at the same memory location, or are small enough integers or floats). If they're copies (as is the case in the above), it returns NIL. If this isn't what you expect, you should probably be using equal.
If that were true I would consider it a very weird Common Lisp implementation. The main point is that you should never depend on it being true even though the two forms look the same.
A couple more examples for the Common Lispers in the audience that highlight the difference between the reader and the evaluator:
> If that were true I would consider it a very weird Common Lisp implementation.
One of those is called SBCL.
'The main point' is that you can't depend on the value of (EQ '(a b) '(a b)) being NIL.
Reason: a compiler compiles a Lisp file. The compiler sees literal data like '(a b) ' and '(a b). The compiler determines that these are 'similar' objects and 'coalesces' them into one object. It might even do it for '(a b) and '(1 a b), seeing that one is a sublist of the other and coalesces the first list into the sublist of the other.
Common Lisp compilers are explicitly allowed by the language standard to do these things with literal data in a source file. It defines a concept of 'similarity', where this is allowed.
$ sbcl
This is SBCL 1.5.8.138-d53fb0eb3, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
* (eq '(a b) '(a b))
NIL
*
Which is exactly what I'd expect in any normal (non-weird) Common Lisp implementation. Is your SBCL returning T for this?
/tmp % more test.lisp
(print (list :eq-or-not (eq '(a b) '(a b))))
/tmp % sbcl
This is SBCL 2.1.9, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (load (compile-file "test.lisp"))
; compiling file "/private/tmp/test.lisp" (written 03 JAN 2022 07:58:54 PM):
; processing (PRINT (LIST :EQ-OR-NOT ...))
; wrote /private/tmp/test.fasl
; compilation finished in 0:00:00.003
(:EQ-OR-NOT T)
T
Excellent example of a place where the file compiler can differ from the REPL, and as you point out this is perfectly legal. It's why #'eq is often not the best equality predicate unless you know what you're doing.
CCL and Lispworks return NIL for this test in both cases.
> It might return true in some implementations (very weird implementations), but in general one should not expect it to do so
That is incorrect. It must always return false. LIST is called multiple times, and each time it is called it must produce a distinct cons. Note this is a different issue from literal coalescing (mentioned else-thread).
A fun exercise when reading posts like those is to mentally compare to TCL. It is somewhat uglier - TCL's lists are multi-valued; and newlines are signficicant, introducing difference between "script" and "command".
Still, the basic eval command looks remarkably similar in lisp vs TCL.
Arguments are evaluated in a different order to modern lisps. It's fexprs. It's weird in tcl because you might see weird macros that are built from the inside out, which is not how they're done with lisp. Like, with syntax-case, you're replacing terms in forms, but tcl does it with upvar and uplevel, so the relation of the form (whitespace matters) and you have to manage items carefully (scoping)
I mean that Lisp's composite type, tuple, always has only two elements. Lists are just syntactic sugar over tuples. This means it is pretty simple to iterate via recursion, without special iteration-related functions -- see _evlist in the post for example.
Compared to that, TCL's composite type is "list", which has an arbitrary number of elements, and recursive iteration is not as simple. Yes, one can use "lindex"/"lrange" to do LISP-style recursive iteration, but this is not idiomatic at all and has horrible performance. Instead, people use foreach/explicit numeric indices, and this needs more support and special iterating-related functions, see [0] for example.
> Still, the basic eval command looks remarkably similar in lisp vs TCL.
The primary difference is whether arguments are evaluated before (Scheme/Lisp) or after (Tcl) being sent to the function/command.
The deferred evaluation is what allows you to do things that you need macros for in other languages. (ie. you need deferred evaluation or macros to implement short-circuit operators, for example)
However, early Lisps had FEXPRs which also deferred evaluation. The late John Shutt built and entire Lisp called "Kernel" around the idea:
If I am reading this right, TCL's approach is neither FEXPRs nor Lisp's macros -- instead, it is a very concise "quote" command and "uplevel" command which is something I have not seen widely used in any other language.
In the lisp terms, imagine a language with no macros and with immediate evaluation ("applicatives" in the Kernel-speak). And no special forms either, except for syntax-level quote command. This means that, for example, for "if" you have to quote the code parameters to make sure they are not evaluated too early:
(if var '(print '(hello world)) nil)
the naive implementation has an obvious problem with variable scope, which you solve with "uplevel" command which evaluates expression in caller's scope. So there really is no difference in the data vs code, syntax-wise -- the first quote in the example above defines the "code" while second one defines "data", and only the function implementation knows how each argument will be used.
And that's how you build a language with homoiconicity and macro-like facilities, but without FEXPRs or applicatives or special syntax. Granted, it has plenty of downside, but it's still a very interesting approach.
In my opinion, there are two big things that Kernel brings to the table:
1) Arguments are not automatically evaluated before being passed to the underlying "operative" in Kernel-speak. This is important because it obviates the need for the macro machinery. This is fairly similar to Tcl.
2) Environments are first-class in Kernel. This is important because you can now constrain the scope explicitly. Do I want to evaluate this in the defining context (lexical scope)? Do I want to evaluate this in the calling context (dynamic scope)? Do I want to evaluate this is global or initial scope? All of these things are possible because environments are a thing that can be passed around.
The downside is that this level of mutability seems to constrain the ability to pre-compile things.
One of the "problems" that I would consider Tcl to have is that certain "atoms" don't self-evaluate. There's really no good reason for why you have to write:
% if 1 {expr 2} else {expr 3}
instead of
% if 1 2 else 3
And no good reason why you can't write:
% + 1 2
3
instead of
% expr 1 + 2
3
This seems like going gratuitously out of the way to avoid "looking like Lisp" which was in its down phase by 1990.
I have the same response to this as I do to every other lisp article. It’s cool, but how have you personally leveraged this advantage that everyone talks about? What big lisp project has the author contributed to, such that they have enough data to sing it’s praises?
I follow that up with knowledge of a large project that was started in Lisp (actually Scheme) that had to be converted to C# because the difficulty of finding experienced Scheme developers for the years of maintenance that would be expected was far more than the cost of converting to a language that's more "usable."
Still remember all the meetings that generally always included someone complaining "what the $#&^%! were they thinking and why didn't their Management stop it before it got this far."
I know a startup that hired a Marxist-collective group of programmers. I was told this story years ago, and they were acquired, so I’ll just name the firm - “White Ops”. This story is so absurd that I’m naming the firm in hopes someone can verify the accuracy of this, although I trust the person that told me the tale.
The programmers were based in Canada and only wrote in Haskell. The CTO of White Ops had apparently hired them previously as contractors at a prior company without issue.
In hindsight, it turned out the purpose of Haskell was specifically because it was very difficult to hire anyone to replace them, and they wrote in a strange manner that left even experienced Haskell developers confused, especially without their customized IDE extensions.
After a couple rounds of funding, the Marxists went on strike and demanded huge amounts of equity from White Ops to continue working. Ultimately the company wound up rewriting everything in a more popular language than acquiesce to their demands. I heard the Marxist group also had tons of drama because the leader made everyone live onsite in shared housing.
There was a furry who worked as a web developer for a small firm. He had built their back-end runtime in MUCK scripting language. (A MUCK is like a MUD, but without the statistics and stuff, so it's much less a role-playing game than it is simply role-playing.) Eventually he was fired, probably because he spent more time pretending to be a highly sexualized female fox on MUCKs than doing his job, but the back end to all their web apps was written in MUCK code -- and few non-furries even knew what a MUCK was let alone how to program one. So the befuddled engineers who remained had to figure out how to modify the code without their vulpine-in-his-own-mind former colleague.
They uh, also didn't notice the backdoor he put into the web server that allowed him to spy on their activities, even though he was gone from the company...
I can't even. Did they use the one that's basically Forth with a string stack (No, not Postscript, Forth with a string stack) or the one that's basically m4 (the higher level of the two sendmail scripting languages) implemented in Forth with a string stack?
I think they used MPI (the m4-like one) because it is much easier to retrofit as a sort of templating language for use in web pages back in the 90s when that was a thing, as opposed to the modern practice of maintaining an entire parallel-universe DOM which is then transformed into the real DOM. MUF (the Forth-like one) was a bit too low-level for the application, though I do seem to recall a story this same furry wrote in which a fantasy CPU is mentioned that runs MUF natively, like a Lisp machine but for furry roleplay environments. I won't get into the uses to which it was put.
SOLID and a dozen of DPs and you get there pretty fast.
Then you rewrite by another team with their interpretation of SOLID and another dozen of "better suited" DPs and you get there again.
On 3rd iteration you just buy something as a service that covers 80% of your needs and consider the rest 20% non-essential business. And life is finally good.
I was told they are self-described Marxists. I am not an expert in the finer points of communist philosophy but I assume this means they believed in the labor theory of value and the class struggle.
As long as “hire experienced X engineers” is a non-negotiable requirement, you better choose a popular value of X.
In parts of the community we do not so much think of the tools as part of the identity of the engineers. We’re looking for general intelligence/capability and train specific technologies on the job.
I don’t consider myself a Lisp wizard and mostly stick to pretty simple code but to name a few cool things that would have been tedious or impossible in another language:
- A desktop application for a bank in which not only the gui’s content but the look and structure was generated or changed on the fly mimicking the status of a database.
- An interpreter in which most of its classes and methods were automatically generated. The Java rewrite took pretty nasty code generation hacks, visitor pattern madness and way more LOC.
- Some program with a huge cache (glorified hash table) that was not catching simple strings or objects as keys but actual code as s-expressions. I always thought that was pretty cool.
Most if not all of my Lisp (and Perl) projects have been eventually rewritten in Java, and not exactly due to technical issues. I mostly put up with Go these days which is great most of the time but of course makes me yearn for a more powerful language or wished I never dabbled with those.
I have implemented pretty much all of that in C++. It wasn’t hard or tedious. Using declarative data structure to runtime generate the UI from (for example) is something I do all the time. In C++, C# and Typescript. And evaluating expressions at runtime is just using an AST. Something I have also done a lot.
The GUI application was particularly elegant because the program would detect the changes in the database, change the class structure and attributes of screens and widgets at runtime with all runtime instances being updated. Note that this was written decades ago, way before people got excited at Swift UI, Qt QML or Javascript web apps.
As for "just using AST", it is tedious to me, that's why I don't bother with it in other languages, and judging by the code that I've seen, it is for the average programmer as well.
Yep nothing that I haven’t done in other languages. I learned Typescript and wrote a 52 page business application from scratch in a few months using the declarative approach. I used JSON to declare everything the same way you would declare it in Lisp and had all the same advantages. Parsing JSON in Typescript is a single line of code. The same as calling read in Lisp.
And a college of mine took the application and easily made a version for another customer in less than a day. Changing most of the 52 pages to match the other customer. The look, the page layout, some of the logic etc. He never looked at the source code.
And mind you I have written my own Lisp dialects just for fun, and have written code in Common Lisp and Scheme. I like the simplicity of Lisp the same way that I like the simplicity of Forth. But I just prefer other languages. Not having types is tedious to me.
That's really cool, but you should then be aware that what you are describing is a subset of what homoiconic languages offer as "data is code".
Some time ago I wrote a sort of DevOps tool in Python (because I know better than to start new projects in Lisp in most settings). This tool had preconfigured tasks defined as point-free style functions that were easily composable. Non programmers were able to define their own pipelines either submitting a YAML file or dragging & dropping these components in a web UI. This was very powerful, but anything remotely complex (if you didn't want to end up reinventing Forth) required dropping down to Python to define these.
JSON and YAML are very good at conveying structure, but fall sort for anything else. It would be like doing any sort of complex programming with Ansible via just using roles and playbooks. It's possible, but no one sane does.
Now the question is if this really matters. Judging at the shape of the industry, the clear answer is no.
Yep there are cool things you can do in Lisp that you can’t do in other languages. But, as you say, it doesn’t matter to most developers. What matters to most developers is a large language community, and a great selection of battle proven frameworks/libraries to make their job easy. Look at JavaScript going from the ugly duckling of languages that nobody use, to being absolutely essential to millions of programmers everywhere. Not because of the language, but because of the large community of developers forced to use JavaScript, resulting in a huge supply of frameworks and support/help available to JavaScript developers. If Lisp had been used instead of JavaScript, it might have been equally popular today. I personally would have preferred that actually.
I'm not a hardcore LISPer anymore, but I worked on a ~600K LISP system in my first job out of college for 4 years. I'm many years away from it, but I still remember it fondly.
We were working on a DoD project to feed relatively large amounts of logistics data in and out of a bunch of simulation models written for different contracts for different branches of the military. The inputs/outputs were all relatively large/complex files, with different formats and different units, and some values had to be repopulated in the output-files before being fed into the next simulation, and the input/output formats of many of these things were often being changed. The whole environment we lived in was more dynamic than you'd ever really want, yet I remember it feeling really easy to describe what I wanted to do in code; more than any language I've worked in since, stuff tended to Just Work the first time.
Some of the cooler things one could do in LISP that were difficult/impossible to do in other languages:
- Running in an interpreter from your editor (emacs, of course) allowed you to write editor code that could read/rearrange/fix your application code in the running image making the tweak/reload/test cycle trivially fast, even in really complicated systems (Python comes close, but not quite there).
- If it made sense, you could write lisp code in your editor that helped you write/manage the LISP code for your project, create patches automatically, etc.
- Writing a refactoring script in your editor was often better than trying to do it by hand. Decades later, I have yet to see an IDE or a refactoring tool that works as cleanly.
- The LISP macro system made it "easy"[1] to avoid writing boilerplate code. It was often more practical to write a boilerplate-generator, rather than thousands (on 10k) lines of boilerplate.
- Since classes/objects could be redefined on the fly almost arbitrarily, it was possible to redefine object data members as static class-members, and subclasses with overridden member-data, depending on the contents of your dataset AFTER loading it (at the time, I think this had a RAM savings of ~30%?)
- CLOS is the cleanest, most flexible object-oriented language I've ever seen, with a sensible multiple-inheritance system and all sorts of tools (like "before", "after", and "around" methods) you could use if you had a good reason to.
- The metaobject stuff in LISP is deep, dark magic; but sometimes you actually need deep, dark magic. Not often, but sometimes.
In the end, I think LISP is still a great choice for really complicated, dynamic problems (the kind that take years and millions of lines of code to solve). If you find yourself working on the sort of problem that makes you feel a need to build a domain-specific language to solve it, that's the sort of problem that LISP is REALLY good for. While I'm not a manufacturing engineer, LISP feels a bit like the programming language equivalent of a CNC mill/lathe that you can use to build out a factory that might require building bigger/better CNC mill/lathes.
[1] "easy" in that the non-boilerplate code was easy to read and understand its intent, but often it was much harder to debug, as macros often called macros which called macros. Debugging code that writes code is a whole different level of brain-hurt, but sometimes it's still better than the alternative.
People complain about the parens; truth is you have a lot of parens in most languages, just not so much concentrated at the outer edges but more interspersed.
The only thing you would need to give up to drop them altogether is &key/opt/rest args.
LISPs power is it's problem. Why would I want a powerful language?
I want a language that has one and only one obvious way to do things. That doesn't require building my own language features or choosing an addon that might or might not be compatible with others.
I don't have any use for the flexibility to customize a language because I want a language that doesn't need to be customized.
If I was doing cutting edge AI i might feel differently, but I'm not. The stuff I do has only one place for creativity and that's the UI. The code should be boring and utterly forgettable. If someone remembers your code, it's probably because they had to spend time looking at it when they could have been writing unit tests or moving on to new features.
Enlightenment doesn't come from actually using LISP, I've concluded, after years of feeling dumb about it. It comes from understanding how LISP is built. Once you understand how LISP is built, you'd never actually use it for a serious project.
My key take aways from learning how LISP is built: 1) Code & data can be unified into the same thing. This isn't particularly deep if you know the concept of an AST. 2) You can bootstrap a language from a simplified version of the language. Repeat this a few times and you get a fancy language. This isn't particularly deep if you've built any compiler, or if you've built LaTeX. 3) You can build beautiful things if you disregard the grittiness of reality. LISP offers an abstraction of physical computers that does not at all resemble a real computer. Some people like this abstraction. Maybe because they don't like how computers actually work.
You clearly don't, since your aim is to write boring and forgettable code. I don't think there is anything wrong with that -- you can write boring code to solve interesting and worthwhile problems, and overall the tendency is probably to err on the side of getting distracted by hip or shiny tech. But some people enjoy writing interesting code, and sometimes interesting code is needed to solve interesting problems and a powerful language can make certain types of interesting code much less painful to write. Auto-differentiation might be a good example.
That's definitely fair, there does seem to still be plenty of worthwhile problems that require interesting code.
It's mostly way above my level though, so for now I'll stick to solving the trivial stuff, so the real geniuses can get back to work.
Nobody might ever say "Wow that's some great code", in fact they'll probably say it's bloated and mediocre... but they'll be glad I thought to put an override button for that one edge case of the super trivial thing they thought was fine!
> Eric Raymond went so far as to say that understanding LISP is a “profound enlightenment experience.”
Maybe this is true for language hobbyists or whatever, but every few years throughout my career, I went back to LISP or LISP-like languages and never found it super profound. Do I need to spend more time on a project? I just don't get the same excitement others seem to get.
It depends on your interest. For me in particular Lisp was a "profound enlightenment experience" because:
1. I have been programming for a long time and had so many problems I could not(in practice) solve in any language but Lisp.
2. Lisp gives you two things no other language gives you: very basic Metaprograming(it can read itself) and the ability to design your own DSL(domain specific language).
3. It also gives you REPL(Read eval print loop) immediate feedback other languages tried to copy(but have not been able to replicate entirely).
Having said that, Lisp design is also obsolete today because it was mainly designed in the 1970s. I hate it (specially parentheses and interface) and love it at the same time.
>Do I need to spend more time on a project?
You don't need to do anything, nobody needs to learn a new language for living.
The fastest way to understand what Lisp is all about is to be near a very proficient Lisp programmer, because he will be able to run circles around you solving problems.
That combination of someone proving to you what is possible at the masters level and natural human competition is probably much more powerful than anything else you could do on your own.
We are humans after all, we see a great hunter hunt and only then we desire to be as good as he is and learn from him. We do it subconsciously and that is the reason elite tennis players sell razors so well, people copy naturally those who they admire without being conscious about it.
Some of it is because a lot of features that used to be unique in Lisp (or few other "rare" languages) became way more available in other languages - partially thanks to Lisp&co popularising them first.
Other stuff is that it tends to take some time working, and actually ending up using some of the more interesting bits, to see effect.
Lots of people for example talk about macros when it comes to homoiconicity, but rarely (outside of old lisp wizards) do I see people going further to how Lisp makes it easy to build languages and compilers that are easy to debug and handle - I'm not just talking about the common idea trope of "write DSL for your program", because a) you're going to write a partial language specific to your program anyway, no matter the language, you might just obfuscate it calling them functions and types b) the easy manipulation of code and lack of parsing (and thus poof goes half the dragon book) means that writing compilers is simply... simple. And with CL's macroexpand and similar features in other lisps, it tends to be much easier to figure what is going wrong (and it's reasonably easy to add reporting of errors in compiled code down to specific form)
Having only tried out Scheme and Common Lisp (I loved them both, what little I experienced) in my CS classes back in the day, I have always wondered how complex programs in these languages actually look like.
I feel most examples just are the same list-manipulations / Fibonacci- / prime number calculation ad nauseam.
Are there any active projects out there that Lisp family programmers would recommend as a good example of what the language can do and be used for (preferably with common elements such as networking, db connections etc)?
139 comments
[ 0.17 ms ] story [ 3514 ms ] threadIt will give you the basic idea, but it won't tell you about macros, reader macros, compiler macros, and having the whole language always available to build your abstractions on; it won't tell you about CLOS, including multiple dispatch, metaclasses, or method combinations; it won't tell you about the condition system and programming from inside the debugger; it won't tell you about live recompilation and live coding; it won't tell you about the JVM interoperability of Clojure, Kawa, or ABCL; it won't tell you about Racket's tower-of-languages approach to programming; I can keep on going for a while. And, since most Lisp dialects and implementations are nowadays compiled, either to native code or to bytecode, the just-interpreting approach doesn't even work very well.
Articles like this are fun, but they end up completely missing the point. This isn't "the power of Lisp"; it's just the very, very, very beginning. The rabbit hole goes much deeper.
[0] http://www.paulgraham.com/rootsoflisp.html
The best kind of comments are the ones which respond to the what as opposed to the who. :D
The Pascal one was used to build TeX, indeed.
Truly asking for help: can you help explain what can I do with macros that I cannot do with functions? Or, maybe, cannot do with high quality or low complexity using functions?
[1]
Some other languages support similar, if not more powerful, compile-time facilities (such as Lisp macros), but those are outside the scope of this article.
https://en.wikipedia.org/wiki/Template_metaprogramming
When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it.
In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon.
[1] https://www.youtube.com/watch?v=mSgXWpvQEHE
This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct.
[0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated.
[1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL.
Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears.
I'll keep learning. I'm sure the light bulb will go off eventually.
Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros.
Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure:
If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop).But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial.
So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler.So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op.
This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it.
Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does.
Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages.
By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it.
As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages.
I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try.
[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72....
[2] https://www.haskellforall.com/2012/12/the-continuation-monad...
[3] https://vimeo.com/230220635
Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros.
First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation
Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility.
Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp.
It's simultaneously very prosaic and very deep. Macros are pretty simple:
1. Access to the AST of a chunk of code for arbitrary processing.
2. At compile-time.
This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion.
Where Lisps are different:
1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures.
2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation.
3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally".
4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1]
These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build proces...
That's what drives me on. That and Rich Hickey's enthusiasm and the sense that he knows something really valuable that I don't. I watch a lot of videos as well, so I can't remember who said that programming languages were user interfaces between human and machine and are an attempt to talk to machines to get them to do things. In that sense, programming languages are a lot deeper than syntax convenience or efficiency. A good language allows us to talk to machine at a much higher level, and I get from functional programmers that this is what they find so attractive.
I will contact you if I get really stuck and when I finally get it just to share it with someone. Thanks!
edit: I've heard a lot about SICP but never looked into it. I think it's time, now that I have time
Videos: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
Text: https://github.com/sarabander/sicp (links to epub and html versions at the top of the README)
I don't have a computer science degree and I think that has held me back some, but I try to overcome it by always trying to learn and never thinking I should already know something or feeling like I'm an expert. I don't have the patience at my age to go back to school and get a degree. That you attempted SICP a few times is a good sign that you don't give up and speaks to your success in finally understanding.
Compared to Elixir (another language I like) - it is homoiconic but it doesn't have this feature. I can't, for example, take an arbitrary function definition and update or define it in the REPL - because all functions must be defined in a module. However, I like these modules because I think it helps enable discovery of functions - it's much easier to find all operations you can do on lists or maps in Elixir than Clojure.
Also, I wouldn't necessarily call macros more powerful than functions. They're just different, and they enable different things. You cannot, for example, pass a macro around like you can a function. Personally I've never written them, but I use them every day because the core library defines macros. I view them as ways to let people smarter than me to give me tools I can use to write my programs.
As far as driving you on, I don't think Clojure is the end-all be-all language. I happen to like it. I also like Elixir. I know people who really love other, more standard languages though. I don't think you need to force yourself to love or "get" Clojure, but playing around with it can expand your mind a bit, especially if you've never used an FP before.
But if you want an FP that (IMHO) would be easier to transition to, Erlang was the first FP that I "got", afterwhich I tried Clojure again. The syntax is a lot more like traditional imperative programs which can ease the transition. Nowadays I would recommend Elixir over Erlang for several reasons: more modern with modern tooling, less syntax quirks, more active online community, lots of active development.
For me, the main benefits of Clojure are (other FP languages have some of these too):
1. Sharable, immutable state as the default.
2. A set of higher order functions that can do all sorts of data transformation.
3. Being able to use the same language on the web in client + server (without having to use Javascript).
4. Clojure gives me the ability to, more than other languages I've used, target the level of abstraction that I need for the problem at hand, which removes a lot of cruft.
5. Lack of special syntax: helps enable REPL driven development, makes it easier to seamlessly adopt new features from other languages.
6. It's hosted on two ecosystems I'm very familiar with - Java and Javascript.
https://vimeo.com/230220635
A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need.
But they are still normal functions - it's the time of execution that changes.
I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.
At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated.
There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false.
*See pg's _On Lisp_ for an in-depth discussion of macros.
What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)
That's why I stopped doing it, but I totally get that in low level programming that it comes in very handy in some circumstances.
As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary.
Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.
[0] https://coalton-lang.github.io/20211010-introducing-coalton/
1. Clojure does not have all of its various conditionals as language built-ins. There is a single conditional construct built in to the language, if. The rest are all defined in terms of if. Function arguments are eagerly evaluated in Clojure. How do you write a short-circuiting function in Clojure? (when predicate consequent) as a function must evaluate both the predicate and the consequent. If consequent is slow to calculate then your whole when expression is as slow as consequent, even if predicate is false and we do not need consequent. when is a Clojure macro based on the if special form.
2. core.async is a mere library.[0] This implements CSP (same idea as Go's channels) and async programming (the sort of stuff that needs compiler support in other languages such as C# or Go) as a library. If Rich Hickey didn't write core.async, you could build this control flow yourself for async programming. In fact, you can create arbitrary control flow mechanisms using macros. Common Lisp's entire object system can be implemented in macros.
3. The only special forms in Clojure are def, if, fn, let, loop, recur, do, new, ., throw, try, set!, quote, and var. The rest of what you would consider the core of Clojure consists of functions and macros. Much of the control flow functionality is implemented in macros.[1]
The core of macros is that they allow you to evaluate arbitrary user code at compile time. Clojure code is represented as native Clojure data structures (the same is true for all Lisps). The entire standard library which you are accustomed to using to manipulate standard data structures is available at compile time to transform the data structures that represent your code. I can't explain this better than Rich Hickey does in [1], so I encourage you to watch the linked section of that talk.
[0]: https://youtu.be/yJxFPoxqzWE?t=560 See this video for an overview of core.async. Later on he talks about the stuff that is implemented as macros in the library. This link starts up when he's talking about C# style async.
[1]: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get an overview of what macros are used for in Clojure's core.
But macros are a powerful option because it doesn't matter if functions are first class or not. If you are using a language where functions aren't first class objects, a macro system would let Zak implement anyway.
So you are correctly identifying that Python can implement this in an alternate way as a function, but a macro system is sufficient to do this and a whole batch of other tricks even if first class functions aren't available as a feature.
Assume, for purposes of the exercise that someone else wrote some of the methods you're calling and they do not behave this way. You could write your own wrappers, but that's a lot more code to write and maintain than a simple control structure that doesn't care what's inside it.
With macros, I can write a short macro regardless of how the upstream code behaves. Without macros, I can request a change to the upstream code and hope the maintainer will agree to make it behave how I want.
It is also arguable that overuse of macros leads to less readable code since you wind up inventing your own personal language the more you do it, which requires more of a learning curve to work on it.
There's a large advantage to sticking with verbose idioms that are common and built out of a relatively few building blocks rather than building a meta-language which is terse.
At the same time I'm not saying don't have macros, but they should really be reserved for more extensive heavy lifting, not a pile of tiny ones used all over the codebase to make it terse.
A well written codebase should read more like Hemmingway at an 8th grade reading level using simple constructs from the base language which are put together, generally using programming design idioms which are common and terminology which everyone shares. There shouldn't be a lot of "what does this macro do and where it is it defined?" questions on every other line.
In Common Lisp:
That way I can easily implement primitive control structures, without resorting to add to my code visibly complex things (like making expressions into functions).Above is a recursive macro, which takes its subexpressions and returns a simplified expression, reusing itself. That way code transformation itself is described as a recursive process. One can learn how to apply such transformation patterns to the code itself and then it gets relatively easy to write.
There's also an advantage in clarity because an abstraction can be defined. Once you know what first-working is for, the purpose of this block of code is clear as soon as you read the first token. Using a pattern instead of an abstraction, it's necessary to read the whole loop to make sure it's really the pattern you think it is.
firstWorking() ?? getItFromTheCache() ?? getItFromTheDataBase() ?? getItFromAnExternalApi() ?? computeItTheSlowWay() ?? defaultValue;
Perl OR short-circuit:
firstWorking() or getItFromTheCache() or getItFromTheDataBase() or getItFromAnExternalApi() or computeItTheSlowWay() or defaultValue;
Spend 5 minutes playing with -> [0]. Appreciate how pretty it makes the code and how nice it is for laying out sequential operations. What a fun operator. Then try to implement it yourself as a function.
When you discover that you can't, you will have discovered the unique power of macros (note the docs link to source code so you can see the readable 8-line macro implementation). You've also learned a very practical skill, which is how to write threading macros that handle your special edge case fail conditions. It comes in handy from time to time.
[0] https://clojure.github.io/clojure/clojure.core-api.html#cloj...
But I don't think this is a bad example at all, I tend to think of macros as "extending the language/compiler", since they have access to the abstract syntax tree of their inputs, which is what is being done here.
It doesn't matter what specific combination of language features are supported. Macros mean you can implement things anyway. It doesn't really matter what the evaluation logic of the language is, a macro can create syntax that does what you want it to.
If there is a macro system available, a new operator can be implemented in any language. If there is no macro system, it depends what features the language supports.
One fairly simple example is Clojure's `when`, which is an expression that starts off looking like
and gets re-written to before being further compiled. This rewriting happens as many times as necessary until an expression "bottoms out" and can be evaluated in terms of only built-ins.It's instructive to see how much of e.g., Clojure is implemented as macros. https://github.com/clojure/clojure/blob/master/src/clj/cloju... shows pretty clearly the correspondence between the original form and the rewritten form in the definition of `when`. `defn`, which is so foundational to idiomatic Clojure that it's easy to mistake it for a built-in, is a much more complicated example.
Macros can analyse/manipulate the abstract syntax tree of their inputs, so I think of them as extending the compiler.
Now you only need to define the function OR...
It's possible to write pretty complex things like control flow abstractions using just structures and functions/methods, e.g. a CL-esque condition system in Java which only uses classes, static methods, and Java lambdas[0] for its syntax. Possible, but also IMO ugly when compared to the CL counterpart, because the low-level but irrelevant details (such as instantiation via `new` or generics) are still presented to the programmer.
[0] https://github.com/phoe/cafe-latte
I plan to fall down the LISP rabbit hole one day, and continue this series in full detail.
Out of interest where are you working such that you're using Lisp? Pretty sure I've actually asked you before ha, but can't remember. Every-time I get burned out with frameworks and polygot development I end up daydreaming of using a stable language like common lisp for the remaining ~40 years of my career.
I am deeply bound up in Common Lisp because of my history. However, if I may make a suggestion: you might want to take a careful look at Racket: easy to make standalone apps, good libraries, large community, and if you don't like Emacs, the Racket IDE is really pretty good. Sit in on one of the online Racket conferences to see many kinds of cool applications and libraries people use Racket for.
• types let me play with the architecture and interfaces top-down
• the REPL lets me write more complex logic bottom-up
Usually there are a few "hotspots" in the codebase that require a lot of logic and are easier to develop bottom-up; the rest of the codebase is focused more on organization, domain modeling and information flow, where top-down design works really well. In actual practice I expect my mode tends to vary even more than that, but this division seems like a good approximation of how I think about it.
When I write Lisp I usually start testing in the REPL, iterate until the test code gets to be a few lines long, and then copy it over to the package's unit tests. Then I can keep iterating on both the test code and the code being tested.
I'll do something like this in the REPL:
"mylibrary.test" will depend on "mylibrary", so quickload will load both packages when they've been changed, and I can re-run the test quickly using Alt-P to call up the previous REPL input.And the REPL is also really great for "what if" testing that you may not want to keep forever, like, "What happens if I call this function with an invalid parameter?" Hopefully there's a test for that, but it's often a lot faster to just call the function and see what happens rather than go dig through the test code and lose your focus.
C++17 made this minimally useful, and things got spectacularly better with C++20 and constexpr vector/string. 23 is shaping up to move even more of the standard library to constexpr.
I've been a serious Lisp programmer (recreationally and sometimes professionally) for around 10 years and I have programmed in Common Lisp, Scheme, my own weird dialects, Emacs Lisp, etc.
At this point I feel like almost everything written about Lisp is silly. Taken as a language family as a whole, there isn't much that separates Lisp from most of the other languages that are out there _except_ syntax-transformations or macros which, frankly, most people should never use.
For me, the best thing about Lisp is the regularity of the syntax. What I like least about almost all other programming languages is all the useless syntactic doodads and wingdings. The brackets, the indentation sensitive stuff, the unneeded complexity that languages insist on associating with what is and must be the denotation of a tree, in the end.
The other great thing about the language is that you must use `let` or `lambda` to introduce variable bindings and their scope is, therefore, always perfectly clear.
Despite these niceties, Lisp isn't going to make you into a super programmer. It won't really help you solve hard problems. It might help you quickly solve easy problems, but in the end, most of the job of a software engineer is grappling with conceptual problems with which Lisp is only sort of helpful.
As long as I'm holding forth, I will say that I enjoy programming in Scheme _much_ more than I enjoy Common Lisp, which feels incredibly, uselessly, and old-fashionedly complex with all sorts of weird corners to stub your toes on.
Many macros—just like programs as a whole—are bad and unhelpful. But this has little to do with macros, and more to do with programmer technique. It's not intrinsically difficult to write a useful macro, but it does presuppose that the programmer has a useful syntactic idea.
The Lisp family of languages so so broad and loose. It's hard to say anything generally about them anymore. But if you narrow down to Common Lisp, there's still a treasure trove of unique features. Common Lisp and Clojure seem to be the only two lisps where you've achieved broad community agreement on tooling, ecosystem, and general library design. They're also two languages where you can get paid to write them. Can't say the same about Scheme.
Lastly, as a personal anecdote, I don't think I've stubbed my toe on anything in Common Lisp. It's a very safe language with a lot of language features. I have stubbed my toe with CALL/CC though and memory leaks.
I think you're right about CL and CLojure, though. While I really prefer Scheme, its not really appropriate for solving a large set of practical problems.
I have to take issue with this. There are only two languages I consider when faced with a very hard problem: Common Lisp and Haskell. One can solve easy problems in any language. But when I have to solve a new problem from scratch that nobody's ever solved before, I don't reach for Java or Python. I reach for the languages that augment my brain rather than limiting it.
A couple more examples for the Common Lispers in the audience that highlight the difference between the reader and the evaluator:
What should we expect the second form to return? T or NIL?Answer: It might return true in some implementations (very weird implementations), but in general one should not expect it to do so. Why?
Now what should the second form return?Answer: It absolutely must return true. Why?
One of those is called SBCL.
'The main point' is that you can't depend on the value of (EQ '(a b) '(a b)) being NIL.
Reason: a compiler compiles a Lisp file. The compiler sees literal data like '(a b) ' and '(a b). The compiler determines that these are 'similar' objects and 'coalesces' them into one object. It might even do it for '(a b) and '(1 a b), seeing that one is a sublist of the other and coalesces the first list into the sublist of the other.
Common Lisp compilers are explicitly allowed by the language standard to do these things with literal data in a source file. It defines a concept of 'similarity', where this is allowed.
CCL and Lispworks return NIL for this test in both cases.
That is incorrect. It must always return false. LIST is called multiple times, and each time it is called it must produce a distinct cons. Note this is a different issue from literal coalescing (mentioned else-thread).
Still, the basic eval command looks remarkably similar in lisp vs TCL.
Can you elaborate? I am not sure what you mean.
It sucks. Tcl will break you down.
Not original commenter, btw.
Compared to that, TCL's composite type is "list", which has an arbitrary number of elements, and recursive iteration is not as simple. Yes, one can use "lindex"/"lrange" to do LISP-style recursive iteration, but this is not idiomatic at all and has horrible performance. Instead, people use foreach/explicit numeric indices, and this needs more support and special iterating-related functions, see [0] for example.
[0] https://wiki.tcl-lang.org/page/Map+in+functional+programming
The primary difference is whether arguments are evaluated before (Scheme/Lisp) or after (Tcl) being sent to the function/command.
The deferred evaluation is what allows you to do things that you need macros for in other languages. (ie. you need deferred evaluation or macros to implement short-circuit operators, for example)
However, early Lisps had FEXPRs which also deferred evaluation. The late John Shutt built and entire Lisp called "Kernel" around the idea:
http://web.cs.wpi.edu/~jshutt/kernel.html
I strongly recommend a read through his PhD thesis: https://www.wpi.edu/Pubs/ETD/Available/etd-090110-124904/
Unfortunately, his implementation isn't anywhere near as good as his conception. Other people have implemented it better.
http://axisofeval.blogspot.com/search/label/kernel http://axisofeval.blogspot.com/2011/09/kernel-underground.ht...
The Oh Unix shell grabs some of the core Kernel concepts and runs with them in a useful package: https://github.com/michaelmacinnis/oh
In the lisp terms, imagine a language with no macros and with immediate evaluation ("applicatives" in the Kernel-speak). And no special forms either, except for syntax-level quote command. This means that, for example, for "if" you have to quote the code parameters to make sure they are not evaluated too early:
the naive implementation has an obvious problem with variable scope, which you solve with "uplevel" command which evaluates expression in caller's scope. So there really is no difference in the data vs code, syntax-wise -- the first quote in the example above defines the "code" while second one defines "data", and only the function implementation knows how each argument will be used.And that's how you build a language with homoiconicity and macro-like facilities, but without FEXPRs or applicatives or special syntax. Granted, it has plenty of downside, but it's still a very interesting approach.
1) Arguments are not automatically evaluated before being passed to the underlying "operative" in Kernel-speak. This is important because it obviates the need for the macro machinery. This is fairly similar to Tcl.
2) Environments are first-class in Kernel. This is important because you can now constrain the scope explicitly. Do I want to evaluate this in the defining context (lexical scope)? Do I want to evaluate this in the calling context (dynamic scope)? Do I want to evaluate this is global or initial scope? All of these things are possible because environments are a thing that can be passed around.
The downside is that this level of mutability seems to constrain the ability to pre-compile things.
One of the "problems" that I would consider Tcl to have is that certain "atoms" don't self-evaluate. There's really no good reason for why you have to write:
% if 1 {expr 2} else {expr 3}
instead of
% if 1 2 else 3
And no good reason why you can't write:
% + 1 2
3
instead of
% expr 1 + 2
3
This seems like going gratuitously out of the way to avoid "looking like Lisp" which was in its down phase by 1990.
Still remember all the meetings that generally always included someone complaining "what the $#&^%! were they thinking and why didn't their Management stop it before it got this far."
The programmers were based in Canada and only wrote in Haskell. The CTO of White Ops had apparently hired them previously as contractors at a prior company without issue.
In hindsight, it turned out the purpose of Haskell was specifically because it was very difficult to hire anyone to replace them, and they wrote in a strange manner that left even experienced Haskell developers confused, especially without their customized IDE extensions.
After a couple rounds of funding, the Marxists went on strike and demanded huge amounts of equity from White Ops to continue working. Ultimately the company wound up rewriting everything in a more popular language than acquiesce to their demands. I heard the Marxist group also had tons of drama because the leader made everyone live onsite in shared housing.
There was a furry who worked as a web developer for a small firm. He had built their back-end runtime in MUCK scripting language. (A MUCK is like a MUD, but without the statistics and stuff, so it's much less a role-playing game than it is simply role-playing.) Eventually he was fired, probably because he spent more time pretending to be a highly sexualized female fox on MUCKs than doing his job, but the back end to all their web apps was written in MUCK code -- and few non-furries even knew what a MUCK was let alone how to program one. So the befuddled engineers who remained had to figure out how to modify the code without their vulpine-in-his-own-mind former colleague.
They uh, also didn't notice the backdoor he put into the web server that allowed him to spy on their activities, even though he was gone from the company...
I think i know these guys! Their names are Karl, Vlad, and Fred, right? Last I heard they founded the International Common-Lisp Party
In parts of the community we do not so much think of the tools as part of the identity of the engineers. We’re looking for general intelligence/capability and train specific technologies on the job.
https://github.com/codr7/whirlisp
- A desktop application for a bank in which not only the gui’s content but the look and structure was generated or changed on the fly mimicking the status of a database.
- An interpreter in which most of its classes and methods were automatically generated. The Java rewrite took pretty nasty code generation hacks, visitor pattern madness and way more LOC.
- Some program with a huge cache (glorified hash table) that was not catching simple strings or objects as keys but actual code as s-expressions. I always thought that was pretty cool.
Most if not all of my Lisp (and Perl) projects have been eventually rewritten in Java, and not exactly due to technical issues. I mostly put up with Go these days which is great most of the time but of course makes me yearn for a more powerful language or wished I never dabbled with those.
As for "just using AST", it is tedious to me, that's why I don't bother with it in other languages, and judging by the code that I've seen, it is for the average programmer as well.
And a college of mine took the application and easily made a version for another customer in less than a day. Changing most of the 52 pages to match the other customer. The look, the page layout, some of the logic etc. He never looked at the source code.
And mind you I have written my own Lisp dialects just for fun, and have written code in Common Lisp and Scheme. I like the simplicity of Lisp the same way that I like the simplicity of Forth. But I just prefer other languages. Not having types is tedious to me.
Some time ago I wrote a sort of DevOps tool in Python (because I know better than to start new projects in Lisp in most settings). This tool had preconfigured tasks defined as point-free style functions that were easily composable. Non programmers were able to define their own pipelines either submitting a YAML file or dragging & dropping these components in a web UI. This was very powerful, but anything remotely complex (if you didn't want to end up reinventing Forth) required dropping down to Python to define these.
JSON and YAML are very good at conveying structure, but fall sort for anything else. It would be like doing any sort of complex programming with Ansible via just using roles and playbooks. It's possible, but no one sane does.
Now the question is if this really matters. Judging at the shape of the industry, the clear answer is no.
And how close we came... and yet so far =(
https://brendaneich.com/2008/04/popularity/
We were working on a DoD project to feed relatively large amounts of logistics data in and out of a bunch of simulation models written for different contracts for different branches of the military. The inputs/outputs were all relatively large/complex files, with different formats and different units, and some values had to be repopulated in the output-files before being fed into the next simulation, and the input/output formats of many of these things were often being changed. The whole environment we lived in was more dynamic than you'd ever really want, yet I remember it feeling really easy to describe what I wanted to do in code; more than any language I've worked in since, stuff tended to Just Work the first time.
Some of the cooler things one could do in LISP that were difficult/impossible to do in other languages:
- Running in an interpreter from your editor (emacs, of course) allowed you to write editor code that could read/rearrange/fix your application code in the running image making the tweak/reload/test cycle trivially fast, even in really complicated systems (Python comes close, but not quite there).
- If it made sense, you could write lisp code in your editor that helped you write/manage the LISP code for your project, create patches automatically, etc.
- Writing a refactoring script in your editor was often better than trying to do it by hand. Decades later, I have yet to see an IDE or a refactoring tool that works as cleanly.
- The LISP macro system made it "easy"[1] to avoid writing boilerplate code. It was often more practical to write a boilerplate-generator, rather than thousands (on 10k) lines of boilerplate.
- Since classes/objects could be redefined on the fly almost arbitrarily, it was possible to redefine object data members as static class-members, and subclasses with overridden member-data, depending on the contents of your dataset AFTER loading it (at the time, I think this had a RAM savings of ~30%?)
- CLOS is the cleanest, most flexible object-oriented language I've ever seen, with a sensible multiple-inheritance system and all sorts of tools (like "before", "after", and "around" methods) you could use if you had a good reason to.
- The metaobject stuff in LISP is deep, dark magic; but sometimes you actually need deep, dark magic. Not often, but sometimes.
In the end, I think LISP is still a great choice for really complicated, dynamic problems (the kind that take years and millions of lines of code to solve). If you find yourself working on the sort of problem that makes you feel a need to build a domain-specific language to solve it, that's the sort of problem that LISP is REALLY good for. While I'm not a manufacturing engineer, LISP feels a bit like the programming language equivalent of a CNC mill/lathe that you can use to build out a factory that might require building bigger/better CNC mill/lathes.
[1] "easy" in that the non-boilerplate code was easy to read and understand its intent, but often it was much harder to debug, as macros often called macros which called macros. Debugging code that writes code is a whole different level of brain-hurt, but sometimes it's still better than the alternative.
The only thing you would need to give up to drop them altogether is &key/opt/rest args.
I want a language that has one and only one obvious way to do things. That doesn't require building my own language features or choosing an addon that might or might not be compatible with others.
I don't have any use for the flexibility to customize a language because I want a language that doesn't need to be customized.
If I was doing cutting edge AI i might feel differently, but I'm not. The stuff I do has only one place for creativity and that's the UI. The code should be boring and utterly forgettable. If someone remembers your code, it's probably because they had to spend time looking at it when they could have been writing unit tests or moving on to new features.
My key take aways from learning how LISP is built: 1) Code & data can be unified into the same thing. This isn't particularly deep if you know the concept of an AST. 2) You can bootstrap a language from a simplified version of the language. Repeat this a few times and you get a fancy language. This isn't particularly deep if you've built any compiler, or if you've built LaTeX. 3) You can build beautiful things if you disregard the grittiness of reality. LISP offers an abstraction of physical computers that does not at all resemble a real computer. Some people like this abstraction. Maybe because they don't like how computers actually work.
You clearly don't, since your aim is to write boring and forgettable code. I don't think there is anything wrong with that -- you can write boring code to solve interesting and worthwhile problems, and overall the tendency is probably to err on the side of getting distracted by hip or shiny tech. But some people enjoy writing interesting code, and sometimes interesting code is needed to solve interesting problems and a powerful language can make certain types of interesting code much less painful to write. Auto-differentiation might be a good example.
It's mostly way above my level though, so for now I'll stick to solving the trivial stuff, so the real geniuses can get back to work.
Nobody might ever say "Wow that's some great code", in fact they'll probably say it's bloated and mediocre... but they'll be glad I thought to put an override button for that one edge case of the super trivial thing they thought was fine!
Maybe this is true for language hobbyists or whatever, but every few years throughout my career, I went back to LISP or LISP-like languages and never found it super profound. Do I need to spend more time on a project? I just don't get the same excitement others seem to get.
It depends on your interest. For me in particular Lisp was a "profound enlightenment experience" because:
1. I have been programming for a long time and had so many problems I could not(in practice) solve in any language but Lisp.
2. Lisp gives you two things no other language gives you: very basic Metaprograming(it can read itself) and the ability to design your own DSL(domain specific language).
3. It also gives you REPL(Read eval print loop) immediate feedback other languages tried to copy(but have not been able to replicate entirely).
Having said that, Lisp design is also obsolete today because it was mainly designed in the 1970s. I hate it (specially parentheses and interface) and love it at the same time.
>Do I need to spend more time on a project?
You don't need to do anything, nobody needs to learn a new language for living.
The fastest way to understand what Lisp is all about is to be near a very proficient Lisp programmer, because he will be able to run circles around you solving problems.
That combination of someone proving to you what is possible at the masters level and natural human competition is probably much more powerful than anything else you could do on your own.
We are humans after all, we see a great hunter hunt and only then we desire to be as good as he is and learn from him. We do it subconsciously and that is the reason elite tennis players sell razors so well, people copy naturally those who they admire without being conscious about it.
Other stuff is that it tends to take some time working, and actually ending up using some of the more interesting bits, to see effect.
Lots of people for example talk about macros when it comes to homoiconicity, but rarely (outside of old lisp wizards) do I see people going further to how Lisp makes it easy to build languages and compilers that are easy to debug and handle - I'm not just talking about the common idea trope of "write DSL for your program", because a) you're going to write a partial language specific to your program anyway, no matter the language, you might just obfuscate it calling them functions and types b) the easy manipulation of code and lack of parsing (and thus poof goes half the dragon book) means that writing compilers is simply... simple. And with CL's macroexpand and similar features in other lisps, it tends to be much easier to figure what is going wrong (and it's reasonably easy to add reporting of errors in compiled code down to specific form)
IntelliJ has nothing close to it, does it?
Nope, saw nothing for IntelliJ. However if it highlights parens, one can code in there and `(load "myfile.lisp")` in any other REPL. It's a start.
I feel most examples just are the same list-manipulations / Fibonacci- / prime number calculation ad nauseam.
Are there any active projects out there that Lisp family programmers would recommend as a good example of what the language can do and be used for (preferably with common elements such as networking, db connections etc)?