Ask HN: Best Lisp for software development?
Hi HNers, I've built a lot of software in Python and Ruby over the years and recently, I've become interested in Lisp but am unsure which Lisp/Scheme variant to learn for building production grade software.
Ideally, I would like to learn the most modern one i.e. robust package management, tooling, ecosystem, etc. Really the main requirement is that I would like to be able to build production ready software with a Lisp that I can use at and for my company.
I've looked into Clojure and like a lot about it, but don't want to have to dig into the Java world. Is there any alternative out there? Any suggestions are appreciated :)
137 comments
[ 3.3 ms ] story [ 237 ms ] threadBetter to use SLIME or the eli with it ;).
That being said, there are things you can do in the IDE that I don't believe you can do in SLIME or eli.
Racket is good but the toolchain and libraries are far behind Clojure. Especially Cider and Cursive are so great, and for libraries, there are a lot of solid ones like Ring/Compojure, interesting things like Duct, Pathom, Fulcro.
Common Lisp is also good, SBCL/Slime is solid, but it has its own idioms, a multiple paradigm Lisp-2 and feels pretty different from Scheme and Clojure. I found single-paradigm languages like Clojure and Scheme are sufficient to do most things already.
Java-world-wise, you don't have to deal with Java most of the time if you don't want to. There are plenty of Clojure libraries which of course follow Clojure idioms.
The implementation leaks for some dependencies, and it is easy to drop down into the lower language, but you can avoid knowing much about it.
In theory you could but your experience won't be optimal.
Also having done both Clojure and Enterprise Java. I can tell you its not married to Java. Theres plenty of Clojure specific libraries you can use.
Also try out Nightcode. Its coded in Clojure and is very straight forward. Just hold CTRL and it will show you all available commands.
Scala's sbt works in a similar manner.
If not, every package manager works like you describe, you need it plus something like the JVM and it just works (Maven, Gradle, npm, pip...).
Perhaps it doesn't make a huge difference in practice, but it might help in getting everyone on the same version. For node, you need something like volta or nvm.
On the other hand using yarn or npm is almost completely interchangeable for the vast majority of workflows. The only thing one should be careful about with regards to them are project lock files. Otherwise most of the user facing differences are more about form rather than functionality.
So this means that Gradle, Boot, Lein, Maven, and tools.deps all can consume packages from each other seamlessly, and they all publish to the same repositories in exactly the same format.
The only thing that is different are the way you create build steps and build pipelines. As well as the way you declare your dependencies.
That's why there isn't any friction due to having multiple build tools and dependency managers. Because the package manager is always the same: Maven.
And can lein read from your boot file?
Because npm and yarn tag the same input, produce nearly the same output, and can run tasks defined in the same location (the project.json). Does clj tooling do that as well?
https://github.com/RickMoynihan/lein-tools-deps
This isn't to knock clojure tooling, which is good, but I don't think lein and boot are drop in replacements for each other like npm and yarn (almost) are.
https://clojureverse.org/t/has-anyone-written-a-tool-to-spit...
Lein and Boot are full build tools. They do package management, but they also do task automation such as Gulp and Grunt, bundling such as Webpack and Parcel, and scaffolding such as Yeoman or Brunch.
If you take only their package management subset. Then it does take only 15 min to switch from one to the other. Just like Yarn and NPM both publish too and pull from npm repositories, Lein and Boot publish too and pull from Maven repositories.
It is only once you get to task automation that Lein and Boot start to differ. Rewriting your tasks and build pipeline from one to the other will take you much longer. That's also true of switching from Gulp to Grunt for example.
I also want to add that, bundling is similar in Boot and Lein. While the tasks to do so are not, the way things are bundled is the same. Scaffolding is also interchangeable, since both support each others template formats.
You could pick one and switch to the other after 15 minutes reading.
Everything is packaged together in an UberJar or a Jar. The former can just be dropped on a host and ran, it contains everything needed to run. The latter is meant when packaging a library, it includes only your code, excluding all dependencies. Maven is used for distribution, Maven central and Clojars are the main repositories used for open source.
So there's no issue of compatibility or confusion in how to package and distribute programs or libraries.
But you have the choice between a few different frameworks for writing your build scripts and build pipelines. You can even just roll your own (seriously, rolling your own would take less than a week).
Because of this, there isn't any friction, just choice of multiple ways to define your build steps. So you can choose your favorite style, declarative with Lein, functional with Boot, or Unix-like with tools.deps.
Choice is everywhere in Clojure though, there's always a multitude of options for everything which can most of the time easily be combined together seamlessly and made to work together pretty simply. This choice attracts experienced devs, but turns off lesser experienced devs. It doesn't help that everything always continues to work, as backward compatibility is taken very seriously. So choice can be overwhelming, be prepared for that if you journey in Clojure. You'll have to make decisions of your own, and owe up to them.
Plus they plan to deprecate the entire language's syntax in the near future.
Clojure is the way to go. It might not be perfect. But it works for most part.
Full excerpt from Matthew Flatt (as of about a month ago):
> Racket's design and implementation is on solid ground, and from this point, it can continue to evolve and improve in many ways. No matter how Racket evolves, the community is committed to preserving what we have achieved: today's `#lang racket` programs will run in the future, and today's `#lang racket` modules can be used in future Racket programs. At the same time, the current language design may be close to a local maximum, and it's not in the nature of the Racket community to be satisfied with a local maximum.
[0] https://groups.google.com/forum/#!searchin/racket-users/rack...
See how Perl lost programmer mindshare in the 2000s because of Perl 5 development had stalled in the favor of Perl 6.
Racket is(was) a language to do home work assignments. I guess even their very dedicated users would either switch to scheme or will just move to Clojure, with the current situation now.
This is a common misunderstanding around the Racket 2 proposal. Racket is completely different from Python and Perl when it comes to syntax. Racket already supports multiple syntaxes over the same semantics, and is designed for people to be able to build new syntaxes themselves.
IOW, Racket 1 (more precisely: `#lang racket`) and Racket 2 will not be different runtimes, only different ways of writing what is to run. You cannot run Python 2 and Python 3 in the same process, or import a Python 2 module in Python 3 file; similarly with Perl 5 and 6. But a `#lang racket` (less precisely: Racket 1) module can be imported and run in the same runtime and context as a `#lang racket2` (less precisely: Racket 2) module.
If I implement Python in Racket, I will call it Python, not Racket 3. Python is Python, Racket is Racket.
By the way, Perl 6 is supposed to have a compatibility mode with Perl 5 too.
All these things look fine in theory, but in practice, no one bets their production code on things like these.
Racket, the platform, is very unlike common programming languages like Python, Ruby, Perl, JS, etc. And I don't mean the fact that it's a Scheme/Lisp. I mean the Language Oriented Programming [1] features it's built for.
It is a platform for designing and building languages. It's only sensible that it should be able to run all the different languages in the same program. In fact, one can create an arbitrary new language and use that new language all within the same running process. In fact, as you can see from the link I cite in [1], the languages can all look and behave completely differently, and yet all be valid "Racket": put them in a .rkt file and load it up and see.
I can now sort of understand your view. I am not a Racket contributor, though I do use it every now and then, and to me "building a new language in Racket" now means something completely different than what it'd have meant before I became familiar with Racket (the platform).
I think the difference stems from the fact that what Racket calls a "language" (`#lang`, in Racket terms), is actually just a syntax. In languages like Python/Perl/JS, the syntax and the semantics are so closely intertwined they might as well be the same thing, and we call this "single" thing "the language". But neither of those syntaxes are as extensible/user-modifiable as in a lisp, and none of those semantics are available for use with different syntaxes as in Racket.
A "language" in Racket does not change the core semantics of Racket-the-platform, only the syntax available to the programmer in that one file/module, and how that syntax gets translated into core Racket semantics. In this way, `#lang pollen` and `#lang bf` are not any special "compatibility modes" of Racket. In fact, right now, there exist multiple "Racket"s within Racket: there's `#lang racket/base` and `#lang racket` and then there's the type-checking variant `#lang typed/racket`. Neither is a special mode of another, or of the runtime; all are just different syntaxes of the same runtime.
I reckon Racket 2 will similarly be just a different syntax. Or, as we like to call those in Racket terms: a `#lang`-uage.
1: https://beautifulracket.com/appendix/why-lop-why-racket.html
Wrt to supporting new versions of a language Racket has an excellent track record. The language mzscheme (used before racket) is still supported - and is implemented using the language machinery present in Racket.
I wrote my entire blog engine on it - https://github.com/rcarmo/sushy/tree/master/sushy - and then ported it back to Python 3
As someone else here mentioned, Python is where most of the deep learning action is. In the end, I settled in using Python for deep learning and use a thin REST service to use my models from Common Lisp or Haskell.
Another approach I used was to convert saved Keras models to something I could import into Racket.
But back to the topic: Hylang is a very cool project and does let you mix Lisp and Python.
Use Graalvm if you want native quick starting binaries or start learning Clojure via ClojureScript
Yes this is very slick, but still a bit bleeding edge IMHO.
I do know the small "binaries" of Clojurescript via the Closure compiler tree shaking/dead code removal means that run time macros/eval is not available..
Is this the same for Clojure/JVM on GraalVM? i.e. are eval/run time macros disallowed/removed to attain small binaries and fast start up?
Writing the code is really the first stage of commitment to a language or platform. You are going to spend way more time running code than you spend writing it. The choice of platform is therefore more important than the choice of language in a lot of ways, and Clojure runs on the JVM (or some JVM-like solution like Graal) so you better have some JVM experts around.
Clojure is the usual go-to for writing production lisp. You should be able to build apps with leiningen, without needing to understand/use and java (except maybe as a runner for your program, if you build uberjars).
Since I build stuff for the web (and Electron, a bit), I use ClojureScript, and am generally really happy with it. It's all the goodness of Clojure — the lispiness, the killer persistent data structures, the wonderfully well thought out standard library — plus the little bit of stuff I like from the JavaScript world — fast boot time, stuff I make can run on the web, good enough debugging/profiling tools, and all the included batteries of the web platform (for audio, video, React if you want it, etc).
I personally use Figwheel to compile my CLJS and do live-reloading, but if I were starting today I'd probably use shadow-cljs. I like to do my own DOM stuff, but if you want to use React I'd do so via re-frame.
Can you please expand on this? I've been using Clojure as my main PL for the past 3 years, and honestly only once I had to actually deal with some Java code (when I needed to figure out some Selenium stuff).
More generally, I _like_ that I have the option of any JVM library. It’s an enormously broad and deep platform and Clojure is a very pragmatic way of exploiting all of that work.
I guess it depends on what you're trying to build. For majority of Clojure devs, I think that's not the case.
Contrast with ClojureScript — It has the same problems as above, but substitute JS for Java and NPM for Maven, etc. I was coming from a web background, so I could easily tell the JS-world from the CLJS-world. That familiarity helped me tremendously in getting up and running.
The situation for CLJS is even better now with things like Planck and shadow-cljs (and Shaun Lebron's cljs/tool and Cuttle before them), since those tools mean you don't even need to install the JDK to get up and running with CLJS (as it should be, I'd argue). The situation for CLJ is a bit better thanks to the CLI tool. But the problems above could have been well handled by the classic non-technical solution: really, really good official docs. That's been getting better (thanks to Alex), but I'm pretty sure it's still going to be more painful to get up and running with CLJ or CLJS than Ruby, Node, etc., sadly.
I like to use "time to first problem and the experience of its resolution" as my own ruler. Maybe it's an XML problem ("wtf is Maven?") but it's usually not until I try writing a program that I experience the first problem either at compile time or runtime. I haven't been keeping up with Clojure (whose latest version I've heard improved things on the error front) but when I first learned, it wasn't long before I made a mistake and paid for it with a giant Java stacktrace. I knew Java, though, so it wasn't a big deal to deal with it, but some instances were quite frustrating since there was a disconnect between useful messages about source locations when running code via a REPL connection versus launching the program from scratch which involves loading and running named Clojure files. It was rather subpar compared to Python, and especially to Common Lisp, which has useful compile-time warnings on top of a superior runtime error system.
Still I've always thought it'd be foolish to dive into Clojure without some Java background, or at least without the anticipation of needing to get some Java background, so it's interesting to read the contrary on this page...
Now, you still need to learn about a package manager and possibly a build tool. So substitute learning Maven for one of the Clojure ones: tools.deps, Leiningen, or Boot.
But I mean, you had to learn NPM one day as well no? And eventually Grunt or something like it, and WebPack, etc. Learning about a language's package manager and build tool is part of any learning of any language. But yes, if you already knew NPM and using ShadowCljs does mean you save yourself that part.
I'm not convinced that learning about how the JVM loads dependencies, gets packaged as Jars or UberJars, and how to use tools.deps or lein is any harder than what other languages force you to learn. So you're probably right in assuming the main issue is documentation not being tailored to very beginners.
You could always try ClojureScript on Node.JS or even ClojureCLR. That said, I have to ask again why the hesitation to rely on the most battle tested most invested in runtime ever built, which is the JVM?
1. More requirements helps. What kinds of production-grade software?
2. For most purposes, you won't see the real benefits, compared to contemporary alternatives, until you have some experience with the Lisp. Enough experience that you realize that, although you can write in a Python or Ruby style in a Lisp, to some degree, there's other various idiomatic programming that's a bigger win sometimes.
3. Once you have some reasonable level of experience, (I'll assert) there's a good chance you become more productive than you would be with a comparable amount of experience in most non-Lisps, in general. But there are some big exceptions to that, such as if you need to do something that can be solved with an off-the-shelf library that's already easily available in Python (e.g., a wealth of ML stuff), for example, and you'd have to write that or an interface for it in your Lisp. Writing it yourself often means you understand it better, and can make it do exactly what you need (including possibly big advantages), but it's an investment.
4. Pick a language and a particular implementation. and make an active user community of that implementation one of the requirements. One of the advantages of Lisps is that they tend to have great communities, perhaps because there's usually little money in it, so you get high-powered people who are there for the merits of it, with a lot less noise. But you do want to find one that's being actively maintained/developed, and that has an active user community.
5. Beware that most development hiring right now seems very driven by fashion, "culture fit", and keywords -- a lot less hiring for "smart, and background they can draw upon to pick up whatever new/other thing is needed". So if you get stereotyped as a "Lisp person", or you don't have one of this year's hot stack keywords on your resume, your next job search might be surprisingly difficult. To a large extent, if you used unfashionable keywords, it doesn't matter if you single-handedly had major accomplishments that would normally be extremely impressive to employers. There are very few Lisp jobs, and there have been very few for a long time. There's a niche for CL, but also a wealth of CL gurus already existing that you have to compete with. There's almost no paying work in Scheme, unless you already have the non-Scheme-specific job and can then pick your tools (i.e, professor, startup, or unusual corporate latitude).[1] If you want to move to a Lisp, but might have to job-hunt in a couple years, Clojure might be the best bet, in that regard, at the moment.
[1] For years, AFAIK, I might've been the world's only consultant specializing in Scheme. But I'm shutting down my business, because there still seem to be only a handful of funded projects using it, outside academia (for university consulting, you need a rare PI and situation, for this to be good) and government (federal subcontractor overhead is a huge burden). Most of the few commercial projects using Scheme are one-person, and don't need anyone else.
(Background: I've done serious production work in Scheme for several years, on some important Web server and cloud technical data science stuff. I also played a very small part in a major project in Common Lisp, using SBCL, and bumped into a bit of CL at another job. And lots of production and research work in C, C++, Java, Python, Perl, etc. separate from that.)
We work with big customer who is 90% Java, and we as C# shop have one project with them. It is all time fights to get rid of us. But we are doing good work and deliver so it is hard for them to kick us out.
What are you going to achieve by running Lisp in production at your company? How would adopting Lisp improve uptime, performance, meeting feature deadlines, and being able to hire people?
No language is a magic bullet. I would highly recommend focusing on outcomes, such as increased performance or reliability or developer productivity, and picking a popular language to achieve those goals so you can convince management to adopt it, and hire people who are good at it.
If you really, really still want to do it, management will probably be more open to a JVM language like Clojure than to any other Lisp.
How they can sell the app to management is way outside of the scope here I think.
But don't try to introduce Lisp into your company, unless you are a cofounder of that company. It's a career limiting move. Instead, work on something you're fond of in Lisp during your off hours.
I think this is correct, unfortunately, due to the place in history lisp occupied in the AI winter. As a result, it's looked upon as a failure, and people tend to dissociate themselves from such things, even when there is a body of evidence to the contrary demonstrating its efficacy.
Is that what you were alluding to or am I being too simplistic? I'm just trying to expand on your remark. :-)
So sharpen your Java, C#, and/or Node skills.
The whole "AI winter" meme is just a psychological complex in the mind of a few old-time Lisp coders. It has no relevance today.
A bit old now, and Schemes only, but still worth a quick look.
The various web/xml/xpath/database libraries for Common Lisp are well supported by SBCL (or vice versa) and make for a great programming experience for web things, in my opinion.
McCLIM also continues to improve and is an interesting, if not fully ready-for-prime-time, choice for interactive applications.
Clojure + JVM Ecology best fits your requirements.
The ultimate programming methodology, perfect unity and simplicity ---- [The Pure Function Pipeline Data Flow](https://github.com/linpengcheng/PurefunctionPipelineDataflow)
[1] https://en.wikipedia.org/wiki/Clojure#Popularity
For a free CL, SBCL is very good. However it's a very "classic" kind of a language, with just text mode tools and a unix bias.
For a non-CL Lisp-like, Racket is fun, but their debugging story is not my favorite. But pragmatically speaking, Clojure is probably the better choice. It is based in the Java Runtime, so things don't work exactly like they would in a normal Lisp, but in exchange you get great interop with a ton of Java libraries and tools, which I think is a net positive.
That said, Franz ACL would still be my tool of choice, if non-free is not a deal-breaker. :)
Allegro is the only Lisp that can somewhat compete with Java on the GC story. It’s simply the best Lisp you can use in the server. Even better than Clojure in that regard as idiomatic Clojure tends to run quite poorly and doesn’t allow for much low level tuning.
LispWorks for shipping binaries to clients. Allegro on the datacenter.
Both have "modern" toolchains, testing and CI support, etc., depending exactly on what you mean by that. I suggest whatever you pick to spend some time trying out a few different workflows though (some of which may hail from before you were born yet still feel great) rather than deciding something based on what seems "modern" from your past experience. There was a time when Clojure was "just another jar" to sneak into an enterprise Java app and introduce a bit of sanity. It wasn't "modern" then, lein didn't even exist, but it was still worth doing for a lot of people. Similarly there's a lot of CL code out there that just chucks stuff into a single script-like file instead of doing things "the correct" way (the author might even expect you to use e.g. SBCL with the --script flag, which will do something very similar to what Python does when you pass a filename to it instead of launching it on its own), just like a lot of Python code out there chucked into a single file, but it can still be useful and perhaps appropriate for a particular production task.
For CL I've recommended this booklet before: https://www.darkchestnut.com/book-common-lisp-application-de... It helps you quickly get setup into one (of many possible) modern-feeling professional flows (by Java standards at least, which you'll also find following the default lein flow on the Clojure side) with a suggested project layout, a version-pinned library-dependencies definition file, and a single-step source-to-exe process via a simple Makefile that's little more than a wrapper around a simple shell script which would expose functions like all() or clean() as commands.
To add a bit more after reading some other replies, it's notable that Common Lisp has multiple implementations to choose from (some open source, some not) and if you're following the ANSI standard (or using various trivial-x or de-facto standard libraries) they can all run your code so you can change implementations rather easily later on if you have a reason to. (There's also an implementation that works well when you need to integrate with C++, and of course a JVM implementation if you do decide you want something in that ecosystem.) Clojure has no standard, it is whatever the on-JVM implementation implements, which has in the past changed and may change again (though the creator has ranted against doing that in general, we'll see if usage of "spec" can continue without changes though) and alter your program's behavior. There's a popular on-JS implementation (if you're looking to write production front-end app code I'd highly recommend getting into ClojureScript over any other Lisp) and a less popular on-.NET implementation, they'll both for the most part run pure Clojure code, but there are a lot of non-portable bits.
Everything else, including SBCL, fails a bit short on the overall experience.
The comparison of the need to write C/C++ when using node.js/PHP/python/Ruby is a good one. It almost never happens unless you're doing something very specialized. And better still, the Clojure case is better than many other languages/platforms because via Clojure you can go low-level from Clojure itself (deftype, unsynchronized mutable, etc) without writing any Java code. Again, the cases where you need such low level constructs are very few and far between. Writing your own in-memory data structures comes to mind.