Ask HN: Production Lisp in 2020?
* poor ecosystem of libraries - few gems, most other half-baked
* poor community coordination
* Dependency management limitations with quicklisp
And some specific red flags like:
* poor support for json[6] * poor support for async
* have to restart the server every 20 days because of some memory leak [3]
* hack to tune GC [5]
If you are using lisp in production for non-trivial cases, do these issues still exist? is there a way you can quantify effort is resolving them, and if yes, what is it? and, finally, if you had to re-do your project, would you chose lisp or something else?
[1]: http://www.gigamonkeys.com/book/
[2]: https://leanpub.com/lovinglisp/read#quicklisp
[3]: https://lisp-journey.gitlab.io/blog/why-turtl-switched-from-...
[4]: https://lisp-journey.gitlab.io/blog/why-deftask-chose-common...
[5]: https://www.grammarly.com/blog/engineering/running-lisp-in-p...
[6]: https://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#s4...
219 comments
[ 0.27 ms ] story [ 254 ms ] threadThere are many Common Lisp and Scheme environments. Many have excellent JIT compilers. They all have their relative strengths and weaknesses. So it really depends on your specific use case as to which environment has the best support. For me the biggest factor is that Lisp is very productive for use by a small and competent team. In those situations they tend to create a DSL and evolve the system. PG's essay describes the reason Lisp fails many projects in the section "The Blub Paradox".
There are so many things that can go wrong in a startup, it's not the time to pick a stack or database you know nothing about.
We do use JSON but mainly for the product's Websocket API towards higher level SCADA system. As we have control over both ends there isn't much trouble really.
The project has soft realtime component also implemented with Lisp, and at least Lispworks exposes a substantial degree of control over GC. We however didn't need to tune anything thus far.
The system has perhaps a dozen direct dependencies to the libraries, luckily few of them posed any challenges in use.
Anyway, thanks for the reply.
If you need a low memory consumption then Clojure is also not a great choice.
We've built an ecommerce site on Clojure though and it's been great. :)
It is designed to be hosted, so, a lot of the above doesn't apply equally for ClojureScript (as one example)
Babashka° is worth a look.
° https://github.com/borkdude/babashka
Babashka is a library on top of SCI and is particularly suited to building command line tools. It doesn't need the JVM, starts up in milliseconds, has access to a large part of clojure.core (including core.async) and can be downloaded as a 13mb binary.
https://github.com/borkdude/babashka
SBCL, SLIME, SLY, ASDF, quicklisp, and ultralisp
Other tools and solutions are available, but familiarity with this seemed to be what was required for comfortable use.
For people w/ lisp dev XP, can I ask this: Is the ".emacs/config bankruptcy" issue a general problem? Specifically, when you use so many powerful devices, macros etc; but with little predictable code structure, then further development that sits well with existing code becomes pretty difficult.
A production codebase hopefully merits very different attitudes :-)
It's kinda fine if every library you use is _slightly_ different if you can have a unified interface for how to use them (spacemacs layers system for me).
It is annoying figuring out which invocation of variables sets the tab-width for _this_ new language though...
Also, most libraries in the emacs ecosystem have become more standardised in their interfaces, so things aren't too different nowadays.
But the global variable thing is interesting; Does regular lisp differ a lot wrt snippets having broad, global effects?
The historical timeline is especially interesting because Reddit's cofounders Steve Huffman & Aaron Swartz were alumni of Paul Graham's first YC batch and PG is the author of the well-known Lisp essay "Beating the Averages":
- 2001-04 Paul Graham : Lisp essay "Beating the Averages" [1]
- 2005-07-26 Paul Graham : https://groups.google.com/forum/#!topic/comp.lang.lisp/vJmLV...
- 2005-12-05 Steve Huffman : https://redditblog.com/2005/12/05/on-lisp/
- 2005-12-06 Aaron Schwartz : http://www.aaronsw.com/weblog/rewritingreddit
The takeaway from the Reddit case study is this: Yes Lisp macros and language malleability gives it superpowers that other languages don't have (the "Blub Paradox") -- but other languages also have other superpowers (ecosystem/libraries) that can cancel out the power of Lisp macros.
You have to analyze your personal project to predict if the ecosystem matters more than Lisp's elegant syntax powers.
[1] http://www.paulgraham.com/avg.html
Once they needed to grow by adding developers, they decided to rewrite into a language that other people already knew.
The only "Super Power" is the complete understanding of what your endusers want so that you can grow.
No need to include the "knowledgeable" modifier.
And one guy in comp.lang.lisp wrote a working Reddit 1.0 clone in Lisp in 4 hours:
https://groups.google.com/d/msg/comp.lang.lisp/9WD9-yEaqMs/H...
https://elixir-lang.org/getting-started/meta/macros.html
Most of them arguably qualify as being somewhere between an 80% solution and a 90% solution for the practical problems that lisp might solve with macros.
They're frequently not anywhere near as convenient to use or understand. But that's arguably a good thing in the long run, because it encourages communities to pool resources and work on a coordinated solution that can be shared by the community. My take on the curse of lisp, for what it's worth, is economies of scale. Lisp makes it too easy (and fun) to just keep your head down and work on your own thing, which hinders pooling of resources.
Lisp macros look like Lisp and in manipulating code via macros you can leverage the rest of Lisp and Lisp's ecosystem.
So writing Lisp macros is entirely natural for someone who already knows Lisp. You don't have to learn or use what is effectively a separate language, as you do with every other language that's not homoiconic.
As Alan Perlis once said, "Beware the Turing tarpit, where everything is possible and nothing is easy."
It would just look weird as fuck.
Cue arguments, I guess, about whether Scheme is a Lisp, whatever their standards say.
There's a perceptual litmus test here. Some people see life on earth and see God. Others see the faintest exploration of the design space, an impoverished code base that made it impossibly far. A miracle, perhaps, but not what we'd see in a billion runs of the simulation.
Same with Lisp macros. Their few variants look bolted on. We haven't begun to explore the design space, yet the best macro systems in other languages are copies of what lisp has figured out so far.
There's an 80:20 rule for language emergence. Ecosystem questions aside, a Turing-complete language needs to start by getting one thing right. APL leveraged multidimensional array handling into a general purpose language. Scripting languages gain much of their leverage from nested hash tables.
Concise versions of monadic parsing are exhilarating in their power and expressiveness. If a language was designed so parsing and manipulating itself was its sweet spot, generalizing this parsing to typed trees would be easy, and we'd invent many new programming paradigms. A general purpose language would follow.
Lisp, an implementation of the lambda calculus, is not this language. Lisp is a machine language precursor to this idea, with various macro systems bolted on.
Haskell, an implementation of typed category theory, is not this language either. At the expression level, Haskell is stuck at a few hard-wired pattern matching primitives. Haskell does not make the assumption that everyone has learned to compose arbitrary parsing operators.
These are going to be controversial assertions in a Lisp thread, but I believe
1. C++ Templates + Concepts is a good Macro facility
2. Most of what makes it good is not borrowed from Common Lisp
On the other hand, I agree strongly with this assertion:
> Scripting languages gain much of their leverage from nested hash tables.
I think I said as much in my comment.
> This seems more akin to Common Lisp's defgeneric and defmethod
Except that Templates are statically dispatched and type-checked.
But also one can perform meaningful compile-time computation with them - I have used this in anger to implement stencils and iteration logic for partial differential equation solvers in 1-3 dimensions so I can say from experience that it is not a mere curiosity. And I don't think that CLOS really has any analog for the techniques used to implement std::tuple.
> But macros let you do something else entirely. You can create entirely new syntactic structures, or modify existing functions (see Norvig's PAIP, chapter 9, where macros are used to memoize recursive functions), or any number of other things.
I think the thrust of my comment is that Lisp programmers identify this type-blind AST-mapping functionality as the core of "a Macro System," while users of other languages (with more syntax to begin with) see this as somewhat secondary to the goal of type-checked compile-time computation.
the racket lang team has investigated this heavily.
https://www.youtube.com/watch?v=ABWLveMNdzg
https://www.cs.utah.edu/plt/publications/popl16-f.pdf
https://www.cs.utah.edu/plt/scope-sets/
> Haskell, an implementation of typed category theory
http://math.andrej.com/2016/08/06/hask-is-not-a-category/
In production using SBCL on a customer project we had good results hiring a SBCL maintainer/developer to work past a few issues. From personal experience, I can assure you that LispWorks and Franz both provide incredibly good support for their customers.
That said, if I had to develop Reddit right now, I would not use Common Lisp either.
Clojure?
Or a non-lisp language instead?
Also, quicklisp didn’t exist/wasn’t as widely used then: I’ve done all sorts of Lisp projects now and never had serious issues finding good libraries for standard web development tasks.
What do you mean by this ?
And PyEvalFrameEx is the stuff of nightmares that can't be really fixed because it hardcoded spaghetti that, if rewritten, could subtly break python in myriad ways that would make 2->3 migration a cakewalk.
Oh, and there's no real test suite to verify if you didn't break something subtle in implementation.
While parsing Ruby code is well known to be problematic, there is a very good reason why Ruby can have interchangeable implementation while Python pretty much can't.
Indeed. The "blub paradox" is profoundly arrogant - it presumes not only that people are too stupid to see the advantages of Lisp, but that Lisp advocates are smart enough to know that other languages have nothing unique to offer that Lisp doesn't.
I don't think that the fundamental flaw with that essay is the arrogance so much as the assumption that "language power" lies on a continuum in the first place.
I'm sure that they had a good understanding of their customers, but the language had to have made a difference as well. These days, we're writing web apps in languages like JavaScript and Ruby, not C++, so the difference would be a lot less significant.
The idea that LISP was a secret sauce for viaweb does not hold up under scrutiny.
If you have access to lexisnexis to lookup news articles from the era, it’s easy to see that.
I shouldn't have brought up logic and statically-typed programming as specific examples; my point was more that it's misguided to consider the "power continuum" as one-dimensional, where languages lower on the continuum are "dumb" and languages higher on the continuum are "weird". As you said, a language with good types is not necessarily more or less powerful than a highly dynamic language.
You are right, though, that the Lisp community read the essay and internalized that all other languages are Blub. In my opinion this led them to collectively dismiss, for far too long, Dictionary Literals in every dynamic language which has eaten Lisp's lunch in the marketplace in the last two decades. Objections were raised ("you need homoiconicity for macros!") and everyone assumed they were valid and stuck with Common Lisp's woeful hash table interface and the loop macro keywords, insisting all the while "oh, that's a DSL, you could implement that with Macros"[0].
And then Clojure implemented Map literals and - what do you know - Clojure still has Lisp Macros!
[0] Efforts in this direction thankfully forthcoming: https://github.com/inaimathi/clj
Map literals are 20 lines of Common Lisp [1], via the programmable in CL reader (that Clojure doesn't have).
Richard Gabriel's "Worse is Better" is a lot more relevant about "Lisp in the marketplace" than anything you've mentioned here.
[1] https://gist.github.com/rmoritz/1044553/b2d2d8e4b933cb0f6c3f...
- that code doesn't provide printing for hash tables. Oops!
- breaks tooling: - doesn't provide editing or syntax highlighting support. - confuses any tool that reads all of the files of a project using the Lisp reader.
- it gratuitously introduces a vector syntax that is unnecessary since #(1 2 3) is perfectly fine and standard across more than one Lisp dialect.
- hard-codes the #'equal equality
- doesn't play with the newer ways of registering Lisp read syntaxes, like https://github.com/melisgl/named-readtables
You are not stuck with anything in Common Lisp.
Instead of loop you have lots of options, from the built-in constructs (do, dolist, map...) to very powerful libraries like SERIES and ITERATE.
You are not stuck with the default hash table interface either. Many libraries for data structure are available, you just pick the one that suits you best.
>And then Clojure implemented Map literals and - what do you know - Clojure still has Lisp Macros!
Most of those Clojure features are already available in CL through the use of libraries. Even the ability to easily call Java libraries, by using the Armed Bear Common Lisp implementation.
That original Reddit system was very simple, yet the Lisp source code was of poor quality (it is currently available on the web). When it was released, the lispers at comp.lang.lisp said they could write a clone in a few days. One of them did it in about 2 or 3 hours (true story).
So it wasn't good code. Moreover they, for some strange reason, chose to develop using a different implementation than the one in the server, and things like that.
Then Reddit got a famous name recommended by PG, who was a Python expert. So he rewrote the thing in Python and obviously, being a Python expert, did a good job.
TO address some of your points;
poor ecosystem of libraries: many great libraries, some incomplete. you must not be afraid to look at the source code to understand and edit
poor community coordination: true, there are few large scale community lisp projects. be the change you want to see
dependency limitations with quicklisp: like what? quicklisp works great. if you are talking about loading several versions of the same library at the same time in a single image, this is being researched and developed
poor support for json: no, there is good support
have to restart the server every 20 days: not familiar, I have had lisp servers running for years now
hack to tune GC: no
if I had to redo my project would I choose Lisp? I would choose lisp, yes. there is no more powerful language than this, and believe me, I have tried many
[1] https://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#s4...
With regards to Steve's issues: ask yourself whether they apply to you.
All those options defaults to special (dynamically scoped) variables, so you don't have to pass them everytime.
Encoding prints to the standard output (you can redirect it to an http stream, if you want), so you don't see double-quotes being escaped, but the string is equivalent (modulo spaces):
The defaults in Yason just returns list for vectors, and map both null and false to nil, but this is not a problem: if you expect your field "items" to be a vector, then nil will denote the empty list, if you expect another field "enable-debug" to be a boolean, nil be good enough too. But in cases where you need to handle them differently, the library can help you distinguish among all kinds of value.See https://phmarek.github.io/yason/
That's his opinion. We have a lot of JSON libs, maybe too many. I use "jonathan" and i'm happy with it.
EDIT: He wrote: "JSON support in Common Lisp is a god damn mess. There are an absurd number of JSON libraries and I don't really like any of them."
He doesn't like any of them, but I do like more than one.
Most of my production code is python, but I also wanted to create a dsl so I used hy. You get to enjoy both worlds
XML is totally fine for storing structured dumb data. Don't program in XML.
I would not be so bullish about implementing your own interpreter - as the best obvious choice every time.
Numbers especially are hard to get correct unless you know what you are doing.
The point is complex configurations tend to grow into a poor programming language. They would be far better as a lisp DSL.
So I think GP is correct.
http://www.paulgraham.com/carl.html
I experienced this in both Clojure and Ruby as well. The fact that developers can spend countless of precious time, cycling through a loop of malleability, reforming an already functional product in production. It costs precious time and money, it is expensive. And more importantly, it is non-stop.
Other programming languages like Go are very conservative with feature changes, and imposes hard limitations by choice, the team/company behind it thinks the same way, that a cycle of O(n) is destructive.
And nowadays, you can't expect a developer to stay on a company for so long, so a sureball tooling to hit it hard, and precise, with less time and tinkering around is much needed than ever.
On the future of programming languages, we will see languages being used not because it is malleable, but because it is sure, precise and right for the job.
- I have to use every variable I declare, otherwise it won't compile (b/c golang authors decided this produces better code)
- Official docs confusingly suggest I structure imported files with the root as github.com/ (b/c golang assumes all code will be open sourced)
- I can only import one file per folder
- There is no built in map or reduce function. You're expected to use for loops everywhere because it is more explicit
Once I discovered the last point I noped out.
> - There is no built in map or reduce function. You're expected to use for loops everywhere because it is more explicit
Go is an imperative language, where no implicit magic happens on the background, like auto creation of resources on runtime, every sequence needs to be manually defined by the programmer. "With imperative code, you’re telling your program how to do something. And with declarative code, you’re telling your program what you want to do. - Tyler Mcginnis, React Fundamentals", depends on the programmer, but I like to be in control.
The Clojure language and core library is extremely stable, almost uncannily so. I can't think of any major upheavals it's had for a long time. Transducers from 1.7 maybe? But those fit in with the rest of the core library and didn't cause any issues.
With Lisp, you can create a "Hello World" program in variety of multiple ways. Sure you can refactor it, starting with how to print the letter "H", throw UTF-16 support there as well. At the end of the day, you wasted precious time and resources printing "Hello World".
Doing things in variety does not mean value.
Doing it right the first time means value, now imagine if it's a team.
If you want a full blown Common Lisp experience, with a GUI based REPL, GUI debugger with edit-and-continue, and all the nice stuff you see from Xerox PARC, TI and Genera demos, they are the ones to get.
SBCL has an exceptionally good optimizing compiler now. It probably generates faster code than LW and Franz in most cases.
I don’t really care about libraries. I think generally it’s not very hard to write an ffi binding to a C library and most of the time you don’t need a library anyway. Most of the time you can just write your own library from scratch as you probably only need a few functions.
But maybe this is more a statement of the sort of programs I work on and if you worked on different systems you would want to use lots of external libraries that you don’t control.
I think the biggest issue with CL is how tightly attached it is to an outdated standard. It makes it harder to have things like good Unix support if your interface is through pathnames and streams.
Other thing which may be annoying about CL are the lack of a good way to consistently go from source code to a binary and that compiler optimisations can be unpredictable
Can you talk more about this?
Autotools used m4 because it was the most portable Unix tool, not because it's a good language to write your build scripts in...
I thought they use M4 because they were doing sophisticated preprocessing?
Perhaps rust would have been a better example as there is less variety in build systems.
The method you described is the best one I have found (load everything and dump an image.)
Because it's trivial.
Compilation is almost implicit, when loading a file.
Creating an executable is only one line of code.
The developer experience was great. I get subsecond hot-reload on all plataforms (app emulator/app on-device/web/server).
The same UI components was shared between ReactNative and ReactDOM. All App issues are reproducible in Web, then I fix in web and I'm sure that App is working.
Even SSR was possible, with some tweaks.
Barring that, as a sibling comment asks, what specific framework and/or libraries did you use to achieve the holy grail of sharing-all-the-things? (thought ReactNative required implementing separate UI for iOS and Android).
Sub second hot reload sounds amazing, btw.
For builds and dependency management: Leiningen, boot or deps.edn
For the front-end (clojurescript): Shadow-cljs + reagent
For the backend (clojure): Nrepl and nrepl.cmdline, Ring, jetty + compojure for a REST API
Frontend and backend can share code with .cljc files that work on all clojure dialects. https://clojure.org/guides/reader_conditionals
Both nrepl and shadow-cljs support code reloading through the REPL connection.
You can compile to a jar file and deploy that, build a docker image, or compile to a native executable with graalvm, or deploy on Aws lambda with Cljs+Lambada. Lots of options.
Common Lisp is generally the first thing I think of for any new work I undertake. There are exceptions, but usually my first choice is Common Lisp. It has been for a little over thirty years.
I've often worked in other languages--usually because someone wants to pay me to do so. I usually pick Common Lisp, though, when the choice of tools is up to me, unless some specific requirement dictates otherwise.
The objections you list might be an issue to some extent, but not much of one. Certainly not enough to discourage me from using Common Lisp in a pretty wide variety of projects. I've used it for native desktop apps, for web apps, for system programming, for text editors, for interpreters and compilers. When I worked on an experimental OS for Apple, the compiler and runtime system I used were built in Common Lisp.
I'll use something else when someone pays me enough to do it. I'll use something else because it's clearly the best fit for some specific set of requirements. I'll use it when there isn't a suitable Common Lisp implementation for what I want to do. I'll even use something else just because I want to get to know it better.
I've taken various detours along the way into learning and using various other languages. I like several of them quite a bit. I just don't like them as much as Common Lisp.
The pleasure I take in my work is a significant factor in my productivity. Choosing tools that offer me less joy is a cost I prefer not to bear without good reason. That cost often exceeds the advantage I might realize from using some other language. Not always; but often.
There was once a language I liked even better than Common Lisp. Apple initially called it 'Ralph'. It evolved into Dylan, which, in its present form, I don't like as much as Common Lisp. If I or someone else invested the considerable time and effort needed to write a modern version of Ralph, then I might choose it over Common Lisp.
For now, though, I'll stick with the old favorite. It continues to deliver the goods.
Off topic, but many years ago I had lunch with Larry Tesler (John Koza was also there) and Larry knew of my CL book from the 1980s and pitched me to rewrite it in Dylan. That might have changed my career trajectory.
[1] http://knowledgegraphnavigator.com/
I liked and admired Larry. I considered him a friend. Once I sat with Steve Jobs in my office at NeXT and argued about Larry's virtues. I took the pro-Larry position.
Your lunch was probably around the time I was working on that Lisp OS at Apple. That was Larry's idea, too.
If you don't hear from me, it's not because I don't want to talk with you; it's because I'm forgetful (one reason I like email better than synchronous calls. Inboxes remember things that I forget.)
Knowledge Graph Navigator sort of intersects with some of my running interests--particularly knowledge representation and frame stores.
Ralph used prefix notation, dylan infix notation. Here you can find an implementation of ralph:(2)
From (2): Ralph is a Lisp-1 dialect that compiles to JavaScript. It is heavily inspired by an early version of Dylan (also known as "Prefix Dylan"), as described in Dylan – An object-oriented dynamic language.
Also (3) is about a Lisp to Dylan translator. There is also norvig lisp to dylan translator cited in (3).
(1) https://en.wikipedia.org/wiki/History_of_the_Dylan_programmi...
(2) https://github.com/turbolent/ralph
(3) https://tim.pritlove.org/2003/11/10/converting-lisp-to-dylan...
Ralph was named for Ralph Ellison, author of a different, later book Invisible Man, about being a black man in America.
The surface language was basically Scheme, with some special forms removed and others added or renamed. The type system was basically a simplified Common Lisp Object System; all values were instances of CLOS classes.
The compiler, runtime, and development environment were built on Macintosh Common Lisp. Ralph inherited MCL's design and aesthetics.
Like Common Lisp, it was designed for writing software by interacting with it and modifying it as it ran. That's the essential point.
The project I worked on, bauhaus, was running basically all the time we worked on it. We worked on it by interrogating and hot-modifying the running system, in the time-honored tradition of old-fashioned Lisp and Smalltalk systems.
Later Dylan was not like that. It evolved into just another batch-compiled application language. You edit some files. You batch-compile them. You run the resulting artifact for testing. Rinse and repeat.
That's how OpenDylan is today. That's why it lost me.
It's also what's wrong with some newer Lisps, from my point of view. They aren't designed for building programs by modifying them from inside while they run. They don't know how to handle existing data in the presence of redefined types. They don't provide interactive debuggers and inspectors that can be used to rummage through the memory of the running program, edit it while it runs, and conitnue execution.
Common Lisp provides these features, and has done for decades. Ralph provided them. Smalltalk does, too. If all the Common Lisp implementations in the world suddenly evaporated, I could get along fine with Smalltalk, after a brief period of adjustment. It's probably purely an accident of history that I'm a Lisper rather than a Smalltalker, anyway.
But Dylan doesn't have those essential features anymore, and nothing that lacks them is likely to win me over--not OpenDylan, not Julia, not Clojure or Haskell, or F#. I like all of those languages, but not nearly as much as I like my livecoding.
Later, Slava went to Apple to work on Swift, which turned out to be another mediocrity, lacking any sort of originality / vision and having a firm expiration date. There used to be a time when programming languages focused primarily on end-user empowerment and paradigm-centered computing was the norm (e.g. Seymour Papert's groundbreaking research in constructionism and incremental learning / development). There is precious little left of that now.
Later, as Dylan evolved away from its roots, it made sense to resurrect the old code name to distinguish the Lisp that we used to write a Newton OS from the later non-Lisp that it gradually evolved into.
People did know about it, though. One of the links in a sibling comment is to turbolent's little Lisp that compiles to Javascript, named Ralph in homage to the language I used every day back in the early nineties.
8th I have actually looked at and it's both _not_ particularly interactive (certainly nowhere near as interactive as Factor) and has a very strange license [1] to boot which makes it a total nop for me. It has been criticized before [2].
[1] https://8th-dev.com/com-license.html
[2] https://news.ycombinator.com/item?id=15672361
What is wrong with 8th's license? It is basically a Forth, so extremely interactive...mostly just messing with a stack...not sure about metaprogramming. I think a production license costs you $200 as a one time fee and you have to pay no royalties or anything for apps (better than many tools out there). Upgrading to a later version comes at a discount too. Most of the runtime is proprietary/closed-source if that is what you were saying (def understandable if you had a problem with that). Also, being a 1-developer project, progress is rapid, but I'd be worried about what happens if the maintainer wins the lottery.
Any language with a REPL can be described as "interactive" but that's not what having a strong focus on interactive development means. As Mikel Evins wrote, it's really about the entirety of a language being geared around interactive development.
I can't blame you for parsing it the other way; it's a more likely spelling. You can blame my eccentric father for the odd spelling of my first name, but he inherited the surname fair and square.
It's pronounced exactly the same way as "Michael Evans."
It's been a while, thanks for making it happen!
I 100% agree that it seems like you can't unwind the stack the same way you can in Common Lisp or Smalltalk though.
I don't know whether the others you mention are or not.
A reasonable litmus test is: define some data structure and make a bunch of instances of it. Now redefine the data structure. What happens to the existing instances? Does the runtime automatically notice that the structure definition has changed and offer you a way to update the existing instance to comply with the new definitions?
Another is to write a function that intentionally contains a call to some other nonexistent function, then call it. What happens when you hit the undefined function? Are you offered an interactive session that you can use to grovel around in the dynamic environment of the suspended function call? Can you see all the parameters and locals on the stack? Can you edit their values? Redefine types and pending functions? Can you supply the missing function definition, resume the suspended function call, and reasonably expect it to return a value as if the function in question had been defined all along?
If the answers are yes, then the language runtime is interactive "in the bones", in the sense I mean. If not, then not so much.
We used to keep known good images to work from. Generally a pristine release image, and a reference image with current project code preloaded, and then one or more working images with programmer-specific or task-specific code loaded. I used to save a working image at the end of the day and launch it again the next morning. MCL would remember my state right down to the positions and contents of my windows, so it was a quick and easy way to drop right back into the state of mind I was in at the end of the previous day.
For source code, we used revision control, exactly as you'd expect. In the Ralph days it was Apple's Projector. Nowadays most folks use git.
It's not super convenient to use revision control for dumped images because they're moderately large binaries. Source-control systems are mostly not great with those. So I keep reference images around, treating them like any other valuable binary asset--the difference being that rebuilding them from scratch is pretty fast and easy with today's fast machines.
My standard operating procedure is to start with a single scratch file that constructs a sketch of my naive idea of the data I'm going to be working with, load it, and start poking it to find out where I'm wrong.
As I begin to know what I'm doing, exploratory expressions turn into definitions of functions and data structures. I refactor them out into proper source files and add references to them into a system loader (like everybody else, I use ASDF for that).
Occasionally I might screw up the dynamic state by doing something dumb. Then I kill the session and restart it. The Lisps I use are generally back up and ready for me to continue in less than a second.
When I've made enough progress to want to build a release that I can distribute to testers, I write a builder file. The exact details of that depend on which Lisp I'm using; Lispworks, SBCL, and CCL each have their own idiosyncratic ways of going about it, but a simple build is generally something like a half dozen lines, and a complex one probably isn't much more than a page or so.
The builder file evolves along with the rest of the project.
The same general process applies to unit and integration tests. In fact, I'm pretty much always testing--testing my thoughts and testing what I just wrote is probably the majority of my activity most days. All I have to do to create formal unit and integration tests is make a testing scaffold and then harvest expressions from my scratch files and test comments to flesh it out.
When I release, I tag a commit in the source repo. I might also make a reference image to keep around locally, in case I want to test something against it or hand it to a collaborator.
I work in files a lot, which is typical of Lispers. Smalltalkers are a little more oriented to living in the image. For the most part, files are a serialization format that you use to exchange code and other resources with other sites. Locally, everything lives in your image. Modern Smalltalk systems have their own built-in interfaces to revision control, either to git or, in Squeak or Pharo, to a Smalltalk-specific revision-control system called Monticello.
This book describes an early version of Dylan that was very close to Ralph:
It wasn't exactly the same. Ralph had some extensions that weren't part of the language described in the book. For example, it had a class for representing blocks of memory allocated outside the Dylan heap (that is, "foreign" memory).It used to be possible to find a copy of the book online, but I didn't turn up anything with a quick search.
I think Bauhaus sources still exist somewhere, but I don't know how you'd get them. I asked Apple about releasing them some years ago and they ignored the question.
A few years later I asked whether they were still in Apple's software library, and the person I asked--who, I would have thought, should know about it--didn't seem to have any idea what I was talking about. Maybe they ditched the software library.
Once upon a time there were videos of some of the demos that various folks did of pre-release Newton features. Maybe some of those still exist somewhere; I don't know.
Here's the link again for reference in context:
http://lispm.de/docs/prefix-dylan/book.annotated/
I should have known Rainer would have it.
Even better: he's also archived the design notes:
http://lispm.de/docs/prefix-dylan/design-notes/
I'd probably use SBCL or CCL for everything, if Lispwork' CAPI or something equivalent worked on them. I do use them when I don't need to build a GUI that I can easily port among platforms.
When I wrote a WYSIWIG text editor years ago, I did it with CCL because it was for my own use, and I didn't need it to run on platforms other than macOS. If I wrote a new version of it, I'd try to find a way to make it work across Windows and Linux, as well. (I can't use Lispworks for that because it's violates their license restrictions on distributing something that can be used as a Lisp development environment.)
I use other languages when I'm paid to, or when I want to do something that is a lot easier to do in another language. For example, I recently wrote a prototype of a web-based collaborative editor and publishing platform for shared documents. I chose Couchbase for storage, because it solves the sync problem reasonably well, and I used Clojure and Clojurescript with React and Reframe, because it worked well with the Couchbase Java SDK.
I've been paid to build prototypes, apps, tools, and system software with Lisp, Icon, ML, C, C++, Objective-C, Java, Python, Swift, Prolog, Haskell, Scheme, and other things. If I had a modern version of Ralph, I'd most likely use that. Failing that, Common Lisp is what I like best.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
To be expected, I suppose. It really was my favorite working language ever.
https://www.youtube.com/watch?v=HOuZyOKa91o
The video was made by a Ruby programmer, rather than an experience Smalltalker, so it's more a newbie showing you cool things he's discovered, rather than an expert showing the power of the system.
On the other hand, there's this one:
https://www.youtube.com/watch?v=o4-YnLpLgtk
That one is an expert, Kalman Reti, showing some of what it's like to work with a Symbolics Lisp Machine. (It's not a real Lisp machine; they're all at least thirty years old now; this is an emulator running the old Symbolics OS on Linux).
Go to YouTube and search for Rainer Joswig and you'll find a few video demos of Lisp Machines and MCL and so forth.
Finally, here are a few images from about eighteen years ago, also from Rainer. They show various views of the Macintosh Common Lisp and SK8 environments. It's not an experienced user showing you around, but at least you can get some idea what the environments looked like:
http://www.lemonodor.com/archives/000028.html
[0] hylang.org
Ladder Insurance is a full stack Clojure shop and Watchful has a bunch of Clojure in prod too.
I am a happy LispWorks customer but I also use SBCL (efficient code, easy to package standalone apps) and Clozure Common Lisp (fast compiler for development).
The only problems we are seeing is the significant startup time cost (hence the Go bits, where it matters).
Memory usage also isn't always great, but that's more a JVM problem, as it's not releasing reclaimed memory as quickly as you'd like it to. Fine on servers, annoying on a 16GB laptop.
The Common Lisp problems you outlined don't really apply to Clojure, it's a modern lanaguage and has been stable and versatile for us.
Persistent data structures are really game-changing.