HN Explorer

Ask HN: Learning Lisp, and I have some questions

94 points by maxpert ↗ HN
So this weekend I am learning some LISP. I thought best way would be to get some basic introduction (which I already did), and then build a project. So for practice project I have decided to make a URL shortner web service. I am setting up SBCL as that seems to be the most popular. I have few questions though:

- What are my package manager options? I can see quicklisp and qi. The quicklisp seems to install everything in a global folder. I am looking something closer to npm or bundler that stays local to my project. Qi seems like it. But I am still confused, what are my options here?

- What web frameworks are recommended. I don't need a full fledged framework, I need something micro, that just get's the job done. What are my options here and what's most recommended?

- For storage database I want to use something embedded and fast. SQLite is one option, what are my other options? Are there any "pure lisp" implementations of Key/Value stores?

- For hosting I know I can get it running on a VPS. Are there any other options I am not aware of out there?

I am total n00b so some questions might sound stupid to full time lispers, so bear with me :)

49 comments

[ 2.4 ms ] story [ 82.9 ms ] thread
pedrodelfino
I am also a beginner. I don’t have much experience with web development (I am an undergraduate Applied Math student with a deep interest on computer science). If you don’t mind, I would you suggest you to change your approach to learning Lisp. Get the classic freely available SICP book (from MIT professors) and do some exercises. The classes are also available at MIT OCW. Another good book is “How to Design Computer Programs”, from the guys who created Racket, a Lisp dialect. Maybe, these books will give you a purer sense of the language. Web frameworks are a different thing. In conclusion, if you are trying to understand Lisp, have a go on pure Lisp. Btw, Racket is great, I prefer Racket an Dr. Racket than Common Lisp.
CarolineW
I don't entirely agree with your reply, but I think it's unreasonable that it's being downvoted. I've upvoted it to try to balance that, although it's not clear my vote will count for much, if anything.

I think what you've said is reasonable, although it doesn't really address the poster's questions.

rjsw
I would just store any data in the Lisp system itself, you have hashtables built in.

Why do you need to host it somewhere else ? Just get it running on your development machine.

crimsonalucard
Web applications are typically stateless, with state being managed by an external service such as a database.
pavlov
It’s not a great architectural choice for most web apps though. I think the reason we’re stuck with it being the default is simply historical: the CGI script execution model, rise of PHP, etc.
jjnoakes
The upside of an external service (even an embedded library if you want to go small, like sqlite) is that it handles the hairy persistence problem for you.

If you keep your data in memory, you may eventually want to store it somewhere so you can restart the service, reboot the machine, restore from older state, etc.

If you start managing the disk writing yourself and care about the edge cases (like ungraceful termination of your process or the os) you have quite a bit of work ahead of you. A service or library takes care of all of the right fsync calls on the right entities in the right orders for you.

pavlov
Databases are fine. I don’t like the rigid idea of a stateless web application server that needs to hit the database for everything.

In my view, the web application server’s primary function is to be a smart cache that responds to HTTP requests. The developer’s goal should be to prevent most requests from having to wait on a socket to return data.

spacemanmatt
I would encourage you to consider the odds that a web app developer will write an application that caches better than a database, and why we don't see that pattern happen in practice more?
pavlov
What do you mean by "the odds"? It's quite hard to write an application that doesn't cache better than the overhead involved in hitting a database on every request (building SQL queries, waiting on the socket, marshalling the result data...)

The app developer has a tremendous advantage over the database simply because she knows what the data is. Personally I find that it makes a lot of sense to think of in-memory objects and their layout first, then treat the database as a persistence solution. This was not doable in the '90s because of memory constraints, but today the cheapest cloud VM instances have more memory than what you'd get on a high-end box back then.

You asked why this doesn't happen more often in practice -- I guess there is simply a lot of inertia involved. Many popular frameworks like Rails are designed around the broken stateless model and can't be reset easily. There are also large platform vendors who benefit from the low-performance solution: companies like Heroku are very happy to pretend that scaling means buying more crappy stateless server instances from them.

spacemanmatt
The odds are low. The premise that you will write a write-through cache that is faster and less buggy than say, PostgreSQL, is one I would bet my enterprise against. It is foolhardy to dismiss the labor-years invested in databases.
crimsonalucard
Dude what is the point of state if it isn't persistent? You get non-persistent memory in your web application as a GIVEN. It's part of the language, just declare a variable.
striking
I think databases are a great architectural choice for just about anything. It makes persistence and querying of complex data essentially a solved problem, especially considering "most" web apps have their foundations in some CRUD-flavored thing.
crimsonalucard
This is the ideal architectural choice for web application architecture. It allows for multithreading and multiprocessing in your web application without the need to reason about state across threads and processes in the web application. This is why developing CRUD apps are so easy, the database abstracts all the race condition problems that could potentially occur away from you so you don't have to think about it.
pavlov
What’s the point of multithreading if your app is going to spend 99% of its time waiting for data to arrive over a socket?

For most web applications it’s a terrible architecture if you need any kind of performance. The real advantage is that there is a huge amount of investment made in supporting this type of app servers, so you can build on something like Rails.

crimsonalucard
>What’s the point of multithreading if your app is going to spend 99% of its time waiting for data to arrive over a socket?

What's the point of building an app if this is what it's doing. Obviously this is not the case for the majority of web services. The minimum viable requirement for a robust website is that it can serve requests without blocking itself. You don't get this with a single threaded process.

The reactor model popularized by nodejs and nginx is the alternative exception to the concurrency model of traditional servers. At best the only type of memory you can use the app itself for is caching.

crimsonalucard
Please note when I say stateless, I mean the state does not persist across requests to the server. Within the time delta of serving the request the server is, of course, stateful.
bigmanwalter
I've built apps which load the entire database into memory at boot. It's a great way to speed things up :)
crimsonalucard
They have something that does this, it's called a cache. Please note if you restart your computer, you lose everything.
gkya
- If I'm not mistaken Quicklisp does allow different versions of a package coexist.

- Hutchentoot is very popular. Haven't used though.

- If you want common lisp only, IDK, but Clojure has Datomic. Though wrt databases, I'd rather care about quality, not implementation language, because it's mission critical software.

- NearlyFreeSpeech.net has SBCL and GNU Guile. But it's shared hosting. It should be easy to put an interpreter binary on a VPS kind-of service though. For Heroku there's a pack on github, though I don't have the link.

The only Lisp project that I could further is my emacs.d, because that's what I really use. I think that's key to learning something, having a real use for it. I wish you luck in your lisp adventure!

aidenn0
QL doesn't allow different versions of a system to coexist, but you can have multiple QL installs on a system for the rare cases in which you need two different versions of a system for two different projects.

Qlot does allow more finer grained control, but I've never had a need for it.

s_kilk
If I may ask a related question, how do lispers deal with the absence of a real boolean type in CL? My mind absolutely boggles at the prospect of not being able to tell the difference between null, false, and the empty list.

Am I missing something?

taeric
I find in most cases, I just don't care about the difference.

That said, pretty sure you can find the specific answer if you need it.

juki
Most of the time there just isn't any need to make a difference between the three. There are basically just two common situations where you need to be able to tell the difference:

1. Telling apart optional arguments to a function that weren't given from arguments that were given and were NIL. Typically giving a default value for the argument is sufficient to avoid the problem, but you can also add an extra parameter that tells whether the argument was given.

    (defun foo (&optional (x :default-value xp))
      ;; X holds the argument given by the caller, or the keyword
      ;; :DEFAULT-VALUE if nothing was given.
      ;; XP is T if the argument was given, or NIL if it was not given.
      )
2. Finding an item that may be NIL. Functions such as GETHASH (which looks up a key in a hash table) usually allow you to specify a default value, and they will return a second value to indicate whether something was found.

    ;; Increment the value of KEY in TABLE, or if the key doesn't 
    ;; exist, use 10 as a default, increment that and set the 
    ;; result to the table.
    (incf (gethash key table 10))

    (multiple-value-bind (value existsp) (gethash key table)
      ;; VALUE holds the value if it exists, or NIL if it doesn't.
      ;; EXISTSP is T if the key exists, or NIL if it doesn't.
      )
Notice that in CL additional return values can be ignored if you're not interested in them, so you only need to use MULTIPLE-VALUE-BIND when you actually need it. You could also wrap it in a trivial macro if you're using it a lot.

    (defmacro if-some-let ((var form) &body (then &optional else))
      (let ((somep (gensym "SOMEP")))
        `(multiple-value-bind (,var ,somep) ,form
           (if ,somep ,then ,else))))
    
    (if-some-let (val (gethash key table))
      (format t "Found: ~a~%" val)
      (format t "Didn't find.~%"))
kazinator
You can't make a difference; there is no three: it's all one object.

If you want an empty list that isn't nil, you have to implement your own list type.

If you want a symbol called "NIL" that isn't self-evaluating and Boolean false, you have to use your own package.

dreamcompiler
It can be a problem in advanced Lisp programming where you'll end up reinventing things like maybe monads. But it's never a problem in simple Lisp programs. (After all, C has the same issue with booleans.) Can you be more specific about your concerns?
s_kilk
I'm most concerned about interacting with other systems, like databases, http services, whatever, where the data that's being worked on includes such values as `false`, `null`, and `[]`.

In that context it's hard not to simply conclude that CL would be at a disadvantage.

juki
In that situation you could simply represent `null` as a keyword :NULL, `false` as NIL and `[]` as a vector (or some other non-list datastructure).
dreamcompiler
Exactly. Symbols are first-class runtime objects in CL, so just make up a symbol name for the various null-like values. Keywords are the easiest symbols for this purpose since they're package-agnostic.
diogofranco
This is a legitimate concern - I've built some production lisp services that handled JSON back and forth, and you need to address this specifically. My solution was to hook custom encoders and decoders to `cl-json`, converting null to :null, false to :false, and [] to nil.

Other conversions are possible of course. This scheme has the advantage of allowing you to naturally iterate over JSON arrays as lists in application code, but you have to take care of the fact that :false evaluates to T, by for example writing your own macros for "if".

This might read like a nightmare, but Lisp makes it really easy to customize everything about it self, which can be beautiful, if you're into that.

aerique
It's rarely an issue for me and more often a convenience.

Perhaps because of the surrounding language?

coldtea
>Perhaps because of the surrounding language?

Probably not, since C also has the same issue, and it's not very important there either, while at the same time being totally different as a language from Lisp.

coldtea
>My mind absolutely boggles at the prospect of not being able to tell the difference between null, false, and the empty list.

What's so strange about it? That's how it has always been in C for example, Python didn't have a boolean type either for a good while, and lots of languages are like that.

kazinator
Firstly, we have to consider that there are programming languages (some heavily advocated to the death) in which you simply cannot have a variable, such as a function argument, which at different times during the execution of the program can sometimes hold an empty list, sometimes a false value and sometimes a symbol.

It is actually the same in most dynamic programs also. A given expression, variable or object slot is understood to be a list, boolean flag or a symbol.

The fact that the domains of lists, Booleans and symbols happen to intersect is not relevant and doesn't really even come up. "Oh, this value a symbol too; nice, but we're not using it as one".

The empty list being false keeps a fair bit of unnecessary verbiage out of list manipulating code.

Variables and object slots being initialized to a default value of nil means that they serve as empty lists or false booleans, which is handy. A block of code which requires an initially stack can just be wrapped with (let (stack) ...). stack is ready to be pushed upon.

Really, when we consider everything, there is actually one item that is sometimes a bother: in a nested list structure, there is ambiguity between nil being just a symbol and a nested list. These two objects are equivalent: (a nil b) and (a () b).

If you're dealing with such data in which you need every possible word to map to a symbol that isn't an empty list, use a separate package, so that the "nil" token in the external data goes to mypackage:nil and not common-lisp:nil. (Treating such external data in a separate package is a good idea for other reasons, too).

eatonphil
Practical Common Lisp [0] is a really great (free) book that will help you understand the language better quickly. I read through that with three other guys at Linode in an informal book club and it went well. All of Zach Beane's blog posts are great for starting out with quicklisp, but here is a good one [1].

[0] http://www.gigamonkeys.com/book/

[1] http://xach.livejournal.com/278047.html

jakelaboss
Lisp Programmer here,

- quicklisp is the default for package management. If you need to load a different version of a package you can use the local projects dir. I've never used Qi before, so I really can't say anything on it.

- For a framework I would recommend either Clack[1] (relatively small but featureful) or Ningle[2] (micro-framework).Both support multiple backends including Huntchentoot.

- There are object store databases and serializers like cl-store and cl-prevalence, but if you really want to take full advantage of Lisp I would recommend you use hash tables and reader macros. There's also plenty of mature (by lisp standards anyway) SQL APIs and ORMs like postmodern[3].

- For hosting I use AWS, haven't used anything else so can't say much on it.

[1] https://github.com/fukamachi/clack

[2] https://github.com/fukamachi/ningle

[3] https://github.com/marijnh/Postmodern

scribu
If you’re hoping to use a lispy language professionally, I would suggest looking at Clojure. It’s a much smaller language than Common Lisp and has a very active ecosystem.

Leiningen is the standard build tool: install dependencies, run tests etc.

http-kit gives you the minimum needed for implementing a web server.

There are several key-value stores written in Clojure, but I would stick with SQLite (with the Korma library for nice composable syntax).

junke
- the global folder of Quicklisp works for most purposes. It is built on top of ASDF, which you can configure too. But I would not worry about that first.

- for a URL shortener, Hunchentoot is already enough. Hunchentoot comes with an "easy-acceptor" scheme, which requires only (defparameter server (make-instance 'easy-acceptor :port 8080)) followed by (start server). But maybe you will need to define you own subclass of "acceptor" instead, I don't know.

- For storage, you can use hash-tables and serialize them with cl-conspack.

    (let ((hash (make-hash-table)))
      (setf (gethash :name hash) "YourName")
      (conspack:encode-to-file hash #P"/path/to/hash"))
Note that conspack:decode-file returns a list of values.

You can also interact with databases.

baldfat
I would say that Racket is by far the best Lisp to learn. It has the basically been founded to teach programming and it has been fleshed out into a very strong general purpose language.

Watch these videos on a quick overview of Racket https://www.youtube.com/watch?v=vKjOTdVi99A&list=PLNm0vh7FGM...

You could program your pet project right afterwards. Also the documentation in Racket is better than any other community I have ever found. Clear, concise and uniform.

If I was to direct anyone into learning Racket I would say that 1) Realm of Racket is a great start for the language 2) How to Design Programs (Free online and a great resource)

Racket is just a great ecosystem that makes Deployment a piece of cake. I am a big polyglot but Racket has been my favorite find the past decade. Last time I got excited about a language was Python.

fasquoika
Just a disclaimer because the parent didn't say it: Racket is a Lisp, but it is not Common Lisp. It is far more similar to Scheme, and you will have to make some adjustments coming from Common Lisp. That said, it is indeed one of the best Lisps today, especially for learning
ethagnawl
> What web frameworks are recommended

If you're intent on using Common Lisp, check out: https://github.com/fukamachi/clack

If you're willing to consider Clojure (widely used, actively developed, target platforms: JVM, browsers, CLR), there are _lots_ of options - I've had success using Ring/Compojure in small, web service projects. Check out Clojure Toolbox's "web frameworks" section: https://www.clojure-toolbox.com

Otherwise, if you're interested in Scheme (Guile, in this case), Artanis looks promising: http://web-artanis.com/

[deleted]
(comment deleted)
cat199
focus on syntax / macros / CL packages - maybe make a 2-layer set of macros/functions to translate urls, deal with terminal IO (you can hook up a TTY to a web server simply), and store the results to a simple assq-on-disk affair, the rest will come later. get this, then asdf and gen executable; you can run multiple lib copies in cl packages and don't need them in production anyway, so global folder isn't important; that said you can customize quicklisp install.. write a macro to do this for you for pratice lol :)

the power is in the lisp language & syntax extension, not reproducing 'npm install foo' using native calls and getting frustrated that the ecosystem is smaller

excuse snarky tone, simply typing quickly

cat199
also, if you aren't using commercial lisp, use emacs. really. though it's the editor war, lisp-based-emacsen were written simultaneously with the lisp machines by the same people .. lisp+emacs grew together like c+unix; which is also a good analogy since it is really another world, best to throw out all baggage if you can..
earenndil
You can also use ABCL in order to get access to JVM frameworks.
arnoooooo
Regarding Quicklisp, when I want to have all the dependencies in the project directory, I run sbcl with the option --eval '(setf asdf:central-registry (list #p"." #p"./lib"))'
rcdwealth
For such simple application all you need is LISP implementation.

1. Package manager is not required. 2. Framework is not required, that is simple application, you can have running as CGI under any server. 3. Storage database? Just use the LISP date stored in files, and you can even keep username/password there.

jiyinyiyong
I'm using ClojureScript. I would definitely argue that ClojureScript is the right language you should pick up. It's based on JavaScript ecosystem and all pieces from npm are available. It's the largest ecosystem currently. If you argue JVM is the largest one, then just choose Clojure.

For ClojureScript, what package manager? npm. Also you may need Clojars and Maven for some Clojure(Script) packages. But you can use shadow-cljs to abstract the Clojars part. What framework? search in Node.js community, you can use it in ClojureScript. Which database? check out Node.js community. How to run on VPS? it's Node.js question.