96 comments

[ 2.6 ms ] story [ 164 ms ] thread
I have sometimes wished that "locations" in Common Lisp were real lvalues, rather than an illusion made possible by macros.
CL-LOCATIVES [1] make these first-class values. It essentially turns any setf-able place into a mutable pointer.

[1]: https://github.com/stylewarning/cl-locatives

You know, what's wild is that Lisp machines had kernels and garbage collectors implemented in Lisp too, making heavy use of locatives as raw pointers to implement basic system functions like writing to IO devices and doing garbage collection. It's wild that we all think the primitive -> managed stack (ASM, then C/Rust/C++/etc., then Java/C#/Python on top) is somehow an intrinsic property of computation, but there are multiple existence proofs (of which the lisp machine is one) of whole computing stacks implemented in managed code and working just fine.

Conventional wisdom: "No... no... no... you can't implement an OS kernel in managed code" Lisp machine OS authors: "Hold me beer. [Laughs in 1970s]."

Most Lisp systems, be it Lisp Machines, or modern Common Lisp implementations do the same. The Lisp Machine didn't really have a kernel though, unless you call the microcode that implemented the Lisp Machine macrocode "engine". As for I/O, that was just memory mapped .. you write/read to some magic number.

What is wild was invisible pointers ...

    A forwarding pointer specifies that a reference to the location containing it should be redirected to another memory location, just as in postal forwarding. These are also called invisible pointers.
(Not a member of the "we wall think we can't implement systems in managed code" club -- there are plenty of such systems dating back to before Unix :-) )
Yep. That era was like the Ediacaran biota of operating systems design -- tons of weird and wonderful forms we struggle to even classify in contemporary terms.
I mean yes up to a point, but how do you even maintain such a thing? Like, if your garbage collector was written by a bunch of really smart guys who made sure that it would never allocate during collection by just knowing the internals of your system really well, sure that's great, but then how would you even know when you'd broken that property? And the failure mode isn't great (you presumably get a very occasional deadlock when you just happen to run down exactly the wrong codepath during GC).

Most languages will let you bypass their layering if you want to. But it's usually ultimately a bad idea.

But that would be a language with completely different semantics.

CL still tries to preserve something from lambda calculus.

No, why? Earlier Lisps had locatives while preserving their semantics, and https://github.com/stylewarning/cl-locatives emulates this behavior (albeit with a slight performance hit).

What in Lisp exactly is similar to lambda calculus, though? Lisp isn't even a pure language.

And lambda calculus isn't even a programming language. The argument was like saying "C still tries to preserve something from Touring machine".
Kramer once asked George “do you ever yearn?” Well I yearn. Every time I see a post on common lisp I yearn to have a job where I do nothing but mathematics and realize solutions in common lisp. Unfortunately I have to get out of bed now and deal with janky code written by someone who can only think imperatively.
(comment deleted)
I feel you. I've written quite a lot of Common Lisp over the years and I've tried to show off its capabilities to co-workers, often in comparison to the language used that that job.

Often I just get "huh, that's neat, now let's get back to mining the coalface".

I have a suspicion that most Lisp articles we see on HN reinforce the perception that this is something other people shouldn't take seriously, much less touch.

This particular post is better than most, and seems like genuine interest, and targeted at people already in the fold, which is fine. Though probably no one else is going see it and think "I gotta get me some of that."

The intentional advocacy posts, on the other hand... I usually don't see them appealing well to even the minority of programmers who are amenable to a low-employability platform. While they seemingly help to keep the platform low-employability, by making a weak pitch in the moment that someone was curious/bored enough to look at that link.

Then one day you really wake up, and realize you do have that job. You hop out of bed with a spring in your step, walk into your office, sit down at your desk...

and start debugging a misbehaving compiler macro used everywhere in a 1500-line multi-nested LOOP.

As someone who will probably never get the time or chance to work in CL, I wish I saw more comments like these just to get less FOMO
"All the bad things about Common Lisp" would be a fun blog post to write up one day.
Every time I see a lisp post like this my first thought is “just use the right tool for the job” and lisp is exactly NOT always the right tool for the job.
Ah… these damn pragmatic facts always ruin everything!
If you think that every time, it is never the right tool for the job?
I mean, if you want to be obtuse. Sometimes a lisp is the right tool, sometimes it isn’t.

Claiming that “getting out of bed and programming in anything other than a lisp makes life less worth living” is at best as extremely immature and strange stance to take.

That quote's not in TFA, nor anything like it.
The parent comment I initially replied to, does. Reading is hard.
Sure, if I squint a lot. For how I read it, your paraphrase took it far from the original meaning.
Squinting makes reading harder. Maybe don’t do that.
To be entirely pedantic: if the job is developing a correct mental model for setf, lisp may be exactly the right tool for the job.
There is one use of Lisp that dominates all other languages: rapid algorithm prototyping. In fact, HN was founded by Paul Graham who knew this secret and exploited it well.
Oh, I’m well aware. Turns out HN isn’t successful because of lisp.

Edit: upon further reflection, this whole “lisp evangelism” is pretty tired. It’s not some magic bullet. Idgaf if paul goddamn graham knows how to write code in lisp. That means basically nothing, at all.

libera #perl often comments that part of the zen of perl and actually being good at it is knowing when to use something else.

Sometimes people get offended when we give them a fully working awk/sed one-liner that does the job better but, uh, that's what we would have used ourselves!

(or "that's seriously computation heavy, use C/Rust/Julia/etc. because much though we love the perl5 VM for quick scripts and large-scale OO business logic apps this is not going to work nicely and trying to force perl to do it will be a rathole")

Though I do sort of miss my 'green threads via call/cc in guile sitting atop an ancient but functional perl event loop' system, it was ~20 years ago now and I'm not suggesting anybody else would ever want to use it, but in some ways it was still nicer than modern async/await style event loop driving nonetheless.

(yes, I'm an outlier and should not be counted)

We may be in violent agreement. Handing someone a awk line instead of a lisp binary is exactly what I meant.
Yep, that was very much meant to be a "yes, and" reply.

There's quite the overlap between perlers and lispers because of (among other things) the shared love of 'building the language up towards the problem.'

People have been known to describe bits of my perl code as easiest read by treating it as lisp that incidentally uses a really weird sort of M-expressions and runs on the perl5 VM ;)

I'm a mathematician working in a common lisp shop, and sometimes, I really do feel that I'm living the dream. We're presently hiring, with more of a focus on software engineering than doodling on pure math, of course, but we'd be eager to interview somebody with love for math and common lisp. Email's in my profile.
How did you guys end up with CL?
It looks like CL is all the rage for quantum computing (and some AI companies).

https://github.com/CodyReichert/awesome-cl

Thinking about recent feedback:

- https://blog.funcall.org//lisp%20psychoacoustics/2024/05/01/... (https://news.ycombinator.com/item?id=40233736) (2024) - https://news.ycombinator.com/item?id=33467269 (2022) "Common Lisp was a conscious decision because it allows a small team to be incredibly productive, plus the fact that it's a live image allows you to connect to it over the internet and poke and prod the current state, which has really allowed a much clearer understanding of the data." - https://lisp-journey.gitlab.io/blog/lisp-interview-kina/ (2020)

Very enticing. My work is largely in computer vision and gpu programming. I feel far removed the rare air I once breathed programming lisp.
Funny, the mathematician I know that works at a CL shop hates it, and is eager to transition into data science in a Python shop. From what he has told me, the Allegro IDE is unpleasant and antiquated-feeling, and the code is hard to work with / not of particularly high quality.
The best you can do is bring the principles of lisp systems to more familiar languages. Lots of languages have lexical closures, macro systems, and introspection capabilities. Python is, famously, an "acceptable lisp": https://news.ycombinator.com/item?id=1803815

(Is something like Numba not a spiritual Lisp macro?)

true, Python is a slow and unstable Common Lisp with no interactive debugger, no restarts, no "live image" interactivity, no core images, no binaries, no macros, no compile-time computing, less compile-time type warnings, less functional constructs, and a GIL.
But also less parentheses!
perl has closures, block based scoping, custom keywords (not exactly a macro system but you can achieve similarly awesome crimes against syntax), heavy introspection, an optional condition system, compile time computing, an integrated interactive debugger core with multiple clients, more functional constructs than python, OO modeled on AMOP, and no GIL.

Slow, bizarre in places, no live image though I can at least trivially replace function implementations and add methods to running code, binary building sucks, and there is of course a whole ass list of things that are uniquely awful (assume that the longest list you've seen from somebody who hates the language has 50% overlap with my personal list and that my list is probably longer) but ... it works. For me, at least.

I loved scheme; I keep picking at common lisp getting slowly better at thinking in terms of the capabilities of a system that's image based and deeply condition-system-ed, and I hope to get there eventually :D

Do you have this list typed up somewhere?
Not really, but I'll give you an example and then if you can give me an idea of why you're curious I can try and provide a few more if you think that'd help.

A good one that I think is fairly typical of my annoyances - 'each' - the each keyword uses an iterator attached to the data structure it's iterating, so if you return or throw part way through doing so the iterator is left half way through and the next attempt at iteration will ... not do what you thought it would.

The reason for this is that 'each' was originally introduced back in the mists of time to allow iteration over a dictionary (hash) backed by a DBM file of some sort, and those files only -permitted- a single iterator at the same time.

Of course, people then started to use it for other things, but by that point making it behave a less surprising way for that would've broken existing code that used it for its original purpose (Larry and I chatted about this once and our conclusions mostly came down to 'alas').

Backwards compatibility is a hard master, and perl (and most cpan modules) work very hard not to break it, which leaves assorted things like that lying around.

My 'solution' is to accept this as fact and simply not use 'each', instead iterating over the keys and pulling the values out in the loop body, and honestly that's fine for me but it's moderately annoying having to keep telling newbies they need to pretend the keyword doesn't exist.

Things that freenode/libera #perl have found to be footguns often enough that we regularly teach people not to do that are collected in https://metacpan.org/pod/Perl::Critic::Community so you can find them with static analysis (and yes, of -course- each is on that list :) and might also be enlightening.

(I personally quite like using the behaviour covered by ConditionalImplicitReturn, but that one probably still belongs on the list because I suspect the vast majority of the time it gets used by accident and becomes a footgun with a delay fuse ... i.e. "if you know enough to use this safely, you should also know enough to turn that rule off" applies)

Part of the trouble I'd have producing a comprehensive such list is that so much of this knowledge has been baked into my brain/fingers long enough that there's not really any thought involved in avoiding triggering them anymore - my primary reason for mentioning it was "I am absolutely not blind to its faults, though every language has some somewhere" because I wanted to explain why I like perl overall without sounding like an Evangelism Task Force member.

But yeah, if you want to point me in a direction I'm still happy to dig through my brain and see if I can find you a representative example :)

setf is pretty imperative!
one hard part I had to get over to start understanding what lisp actually is... it's an imperative language. c.f. scheme or clojure.
I think Common Lisp is nicely cosmopolitan. It has great support for whichever paradigm you want to program in, and/or you can build your own.
Like the Burger King of programming languages.
> by someone who can only think imperatively.

The computer thinks imperatively. I prefer to be on common ground.

I misread the title as "My mental model of self was wrong", and was expecting a very different article!
I only realised it didn’t say that after reading this comment.
Me too. I skimmed through the comments assuming they were talking about the Lisp equivalent of Python's self.
(comment deleted)
I don't know why setf always felt hard for me to really grok back when I was starting programming. It has grown to feel extremely natural. Is nice to know that for most "accessor" style functions, you can use those in setf to update their value.

I never looked too hard at how this was accomplished. Fun article!

> It has grown to feel extremely natural.

That's how all of Common Lisp worked out for me. It all feels strange, then after a couple years of hobbyist usage, it all feels natural.

I wish I knew how to communicate that change to folks. There is a lot of history on many of the forms that exist in common lisp. And much of it is far more practical in origin than I expected going into it.
nitpicking, but I am not a fan of the naming. No need of a capital "A" and no need to recall the class name in the accessor names: "var" instead of "a-var" is enough, since yes, we can very well have an accessor "var" for a class "b". "var" is a generic function. So:

    (defclass a ()
      ((var :accessor var)))

    (var *some-a-object*)  ;; => ok

    (defclass b ()
      ((var :accessor var)))

    (var *some-b-object*)  ;; => ok
 
> If we don’t like using car to indicate the head of a list

… then we can use `first` (and second, third… up to 9nth).

---

https://lispcookbook.github.io/cl-cookbook/editor-support.ht...

https://github.com/CodyReichert/awesome-cl

I am with you, I think setf is making code repetitive and more verbose in many cases, however in this particular case, if you see

    (var *some-object* value)
somewhere far away from the class definition, say in another file, can you tell immediately what that code does? Can you understand it directly by looking at the code? Do you define something? Do you set something or do you just read something like a property list (something like rassoc)?

    (setf (var *some-object*) value)
With setf it is immediately clear from the code what is going on. I don't know if that is the best example, just thinking of all the arguments for and against setf :). Personally, I find it makes code more verbose in many situations.
Since I was talking about accessors / generic functions, I don't get the point of this example:

    (var *some-object* value)
because we also use SETF with accessors:

    (setf (var *some-object*) value)
Quick example:

    (defclass person () 
      ((name :accessor name)))

    (make-instance 'person)

    (setf (name *) "me")
> But the names setf synthesises aren’t recognised by defun and defgeneric

Oh, they are. (SETF FOO) is a valid function name in Lisp - even though it's a list, not a symbol.

    CL-USER> (defun (setf foo) (newval) (format t ";; Setting ~S~%" newval) newval)
    (SETF FOO)
    CL-USER> (setf (foo) 42)
    ;; Setting 42
    42
Somehow I read this as "my mental model of self was wrong" and it totally resonated with me.

Then I saw it was about LISP, which I don't know/use and got a little bummed out.

Now maybe my mental model of self is wrong and I need to learn LISP and create a mental model for setf and that will lead me to an awakening of a correct mental model of self?

Sounds like your mental model of the title was wrong. ;)
read the same, I interpreted it as failing eyesight and tiredness.
Same. I was reminded of a quote I like from a Mongolian Buddhist lama:

"It's not that you're not real. We all think we're real, and that's not wrong. You are real. But you think you're really real — you exaggerated."

> This is a method specialised twice: on the type of an argument (A), and the selector used to within the locator (a-var).

This part is wrong: (setf a-var) is simply the name of the method.

Am I only who read title as "My mental model of self was wrong"? And thought it is about psychology, not about Self programming language.
You are not alone.
And yet, you are alone. So very alone.
My basic mental model of SETF was always relatively simple. We were told that in university, many decades ago.

Lisp has traditionally a bunch of different datastructures. Like cons cells with CAR and CDR accessors. RPLACA and RPLACD were the setters, from a time when characters were expensive, so that identifiers were short and abbreviated (RPLACA would be short for Replace CAR). Then there were property lists with accessors GET (-> Maclisp) or GETPROP (-> Interlisp). The setter was PUTPROP.

So for any datastructure we need to learn both a getter and the corresponding setter.

With SETF in Common Lisp we only need to know the getter. SETF knows for each getter a corresponding setter.

So for a datastructure FOO we don't need to know both GETBAR and BARSET!.

The getter is (GETBAR SOMEFOO) and the setter is (SETF (GETBAR SOMEFOO) SOMETHINGNEW). SETF then is a macro which knows how to set the thing.

Advantage: we don't have to learn ancient functions like RPLACA with calls like (RPLACA CELL 42), but we can write (SETF (CAR CELL) 42). Take the accessor and wrap a SETF around it, include the form for the new value.

In reality there is a bit more magic involved.

Wow that is a great description of awful design choices. The purpose of SETF is avoiding learning the language?
Language features are superseded by better ones all the time. Nobody gets language design correct on the first try, especially when requirements change over time and by application.

If you look at the history of languages like JS or B/C/etc. you better strap in for a crazy ride.

The purpose is to avoid learning a large vocabulary in favor of a smaller, more general vocabulary. That is abstraction, and it's a feature of the language, so you can't learn it while avoiding learning the language.
the purpose of setf is to provide an intuitive, powerful, extensible, and simple to use abstraction for an extremely common operation without introducing any new syntax
You should still know about the rplacd function. You can map over it.

E.g., delete all middle conses of an assoc list, gluing the first and last one together:

  [1]> (reduce #'rplacd (list (cons 'a 1)))
  (A . 1)
  [2]> (reduce #'rplacd (list (cons 'a 1) (cons 'b 2)))
  (A B . 2)
  [3]> (reduce #'rplacd (list (cons 'a 1) '(cons 'b 2) (cons 'c 3) (cons 'd 4)))
  (A D . 4)
If someone finds an application for this, drop me a PM. :)

New people learning Common Lisp don't need to know about rplacd, or not right away. Firstly, destructive manipulation of lists should be deemphasized; it is an advanced topic in itself which can produce bugs.

People who advance to system level work in Lisp, like doing work on implementations, will have to know it. rplacd can appear in code that cannot use setf, because it is processed before setf has been boostrapped.

You will run across rplacd in other people's code even if you don't use it yourself; so if you go beyond just writing programs for yourself into collaboration and maintenance of existing code, you can't avoid knowing about it.

If you ever have to debug the code produced by (setf (cdr place) ...), you will likely see rplaca or a similar function.

The point is to make accessing a data structure somewhat "generic". I only have to change the "(car a)" to "(superdupercar a)" and the "setf" propagates the changes along. This is especially important inside macros.

It is also the reason why Lisps can feel so icky. This is a huge hunk of "macro magic" that can be damn near impenetrable when something goes wrong.

The longer I'm at this the more I'm convinced that macros are the reason why Lisp lost. Even if I concede that macros are hugely powerful, every language that leans into something equivalent to macros winds up with inscrutable errors, debugging issues and/or excessive compile times.

If you change (car a) to (superdupercar a), then (setf (superdupercar a) x) no longer works. You have to program the generalized place system of Common Lisp to recognize (superduper ...) as a place, and define what that means; then all is well.

It could be as simple as

  (defsetf superdupercar superduperrplaca)
If you have a function (superduperrplaca a x).
(comment deleted)
The purpose of the place system is to support side-effectful macros where programmers don't have to repeat what they mean n times.
Yeah, that is great idea, and I agree it sounds great on paper. However, some time ago, I have come to a conclusion that it soon becomes quite verbose and tedious.

I think it worked better in the past when people used acronyms and very short names for function and variable names. Today, with self-documenting code style, names are longer for both functions and names. Constantly typing full paths gets annoying quite soon. Code is also more verbose to read and less code fits into 80 columns.

My personal conclusion is that I actually prefer to abstract those away with a proper name like aset, put, and so on, just to make my own code less verbose.

If they really wanted to simplify the language, they should have perhaps removed 'setq' and 'setf' and just kept 'set' in the language but with the powers of setf. Not to mention that users have to learn how to write own setf accessors, unless it is very simple stuff (the extra bit of magic you mention).

As a "simplification" of the language I think they have failed, if that was the goal. That does not mean that 'setf' is not useful. On the contrary, it is a very useful tool to be able to compute and set a "place", just shouldn't be sold as a simplification.

My Lisp education was mostly from Elisp, but this I understood early when I was reading some random code and came across:

    (setf (buffer-string) "hello")
I thought this was just bananas, turns out that Elisp setf simply has a bunch of hardcoded rules for matching getters/setters for even a lot of non-standard data structures like Buffer that are Emacs specific. So the above gets rewritten to:

    (insert (prog1 "hello" (erase-buffer)))
It's not exactly AGI but still very cool.
This is not "hard-coded" as I would describe it. It is defined in `gv.el`:

    (gv-define-setter buffer-string (store)
  `(insert (prog1 ,store (erase-buffer))))
Unfortunately, it is marked obsolete since 29.1. The NEWS says:

* Many seldom-used generalized variables have been made obsolete. Emacs has a number of rather obscure generalized variables defined, that, for instance, allowed you to say things like:

    (setf (point-min) 4)
These never caught on and have been made obsolete. The form above, for instance, is the same as saying

    (narrow-to-region 4 (point-max))
One interesting thing one can do with SETF is define expanders that dive into accessors on "pure" datatypes, copying as needed.

Suppose we have a pure cons type, call it PCONS, with accessors PCAR and PCDR. Then, the form

(setf (pcar (pcdr x)) y)

turns into (effectively; the actual expansion has temporary variables to avoid duplicating forms)

(progn (setq x (pcons (pcar x) (pcons y (pcdr (pcdr x))))) y)

We need nsetf! :-)
setf is like referring to a mutable reference in C++ or Java.

https://www.lispworks.com/documentation/HyperSpec/Body/m_set...

It seems that OP didn't know what a reference is.

> setf is like referring to a mutable reference in C++ or Java.

Setf is rather a computation of a reference, than a reference.

A reference in Java and C++ is a pure pointer, with some syntactic sugar in java (you skeep */-> to define and dereference it), and few corns of sugar spread on top of it in C++ (can't be null).

Curious writing an article about how setf works and not mentioning 'defsetf', only setf methods. I suspect the gentleman's mental model of setf remains slightly wrong.
> You can’t write code like this: (let* ((l (list 1 2 3)) (h (car l))) (setf h 23))

I think that's the crux, does anyone have an argument for why l should be (1 2 3) instead of (23 2 3)? It breaks referential transparency that you have to substitute the value of h into setf to get it to work.

I also once thought that the magic underlying SETF has access to true L-values, and had also been surprised by it being macros under the hood.

Here are my thoughts on the article:

Firstly, there are errors in the definition of OUR-SETF macro:

1. (symbol-function ,our-setf-function-name) will signal an unbound variable error. ,our-setf-function-name must be quoted: (symbol-function ',our-setf-function-name)

2. Arguments to APPLY are ill-formed. Instead of CONS, LIST must be used.

    (apply (symbol-function ',our-setf-function-name)
           (cons ,new-value ,@(cdr locator)))
Using CONS, (our-setf (head (list 1 2)) 0) expands to:

    (apply (symbol-function '|(our-setf head)|) 
           (cons 0 (list 1 2)))
Which is equivalent to (|(our-setf head)| 0 1 2), clearly not what we want.

Furthermore, an OUR-SETF that accepts multiple places will fail. Consider an example from the article:

    (our-setf (aref a 23) 0)
It expands to:

    (|(our-setf aref)| (cons 0 a 23))
Clearly, an error.

The correct usage of APPLY is:

    (apply (symbol-function ',our-setf-function-name)
           (list ,new-value ,@(cdr locator)))
Alternatively, use a FUNCALL:

    (funcall (symbol-function ',our-setf-function-name)
             ,new-value ,@(cdr locator))
Also, SYMBOL-FUNCTION can be dropped, as both FUNCALL and APPLY accept a function designator.

    (funcall ',our-setf-function-name ,new-value ,@(cdr locator))
This concludes the errors part.

Secondly, I was really confused by the symbol generation for OUR-SETF example. I thought of function-defining macros, such as DEFUN and DEFMETHOD, and they accept lists of the form (SETF X) and not symbols whose name looks like a list. This latter notation could be better explained by using multiple escape characters from the Common Lisp HyperSpec. For example, the bar character: |(our-setf head)|.

Multiple escape characters also mean that symbol generation is needed only in OUR-SETF macro. Generic functions and methods can be defined directly:

    (defgeneric |(OUR-SETF HEAD)| (new-value place))

    (defmethod |(OUR-SETF HEAD)| (new-value (place list)) 
      (rplaca place new-value)
      new-value)
This would also require changes to OUR-SETF macro because the symbols used to name generic functions and methods are now interned in a package.

    (defmacro our-setf (locator new-value)
      (let* ((selector (car locator))
             ; use the selector's package,
             ; selector must be interned
             (our-setf-function-name (intern (format nil "(OUR-SETF ~a)"
                                              selector) 
                                     (symbol-package selector))))
        `(funcall ',our-setf-function-name
                  ,new-value ,@(cdr locator))))
With this change we can even use selectors from other packages.

As the author said, these are symbols with weird names. So we can remove most of the weirdness with more macros:

    (eval-when (:compile-toplevel :load-toplevel :execute)
      (defun selector-symbol (selector)
        (or (get selector 'our-setf-name)
            (gentemp (string selector) (symbol-package selector)))))

    (defmacro defgeneric-setf ((selector &rest selector-params) (new-value))
      (let ((name (selector-symbol selector)))
        `(progn
           (setf (get ',selector 'our-setf-name) ',name)
           (defgeneric ,name (,new-value ,@selector-params)))))

    (defmacro defmethod-setf ((selector &rest selector-params) (new-value) &body body)
      (let ((name (selector-symbol selector)))
        `(progn
           (setf (get ',selector 'our-setf-name) ',name)
           (defmethod ,name (,new-value ,@selector-params)
             ,@body))))

    (defmacro ...