HN Explorer

Ask YC: Ruby and Lisp devs, please answer this (Not a flame war)

27 points by Readmore ↗ HN
I'm a Ruby guy, I've fallen in love with the language thanks to Rails. I have a Master's in CS and in college we studied mainly Java, along with some Lisp and C, but I try to do all my current programming in Ruby.

In an effort to not be a Blub programmer I have taken another look at Lisp to see what I'm missing, but I'm afraid that I can't really see it. I found this page http://innig.net/software/ruby/closures-in-ruby.rb which explains Closures in Ruby and after looking through that I don't really see what I could do with Lisp that I can't do with Ruby.

Again, I'm not looking for a "OMG Python is better than both" I would just like some honest answers about what the differences are. It's always worth it to learn new languages but would it be a better use of my time to start over and learn Lisp or just learn how to proficiently use Closures in Ruby?

Any comments, questions, etc. would be appreciated.

67 comments

[ 2.1 ms ] story [ 83.8 ms ] thread
apgwoz
Your likely to get many comments about macros, so I suggest you take a closer look at what macros actually can do for you. That should be a sufficient enough reason to take a closer look at Lisp (be it Common Lisp or Scheme).
Todd
That's it, in a nutshell. The macro facility is what differentiates Lisp from other dynamic languages that have, seemingly, all of the same attributes--dynamic/lexical scope, closures, etc.

They allow you to write a true DSL. Paul Graham pretty much wrote the book on the subject (On Lisp) which he graciously offers as a free download since it's out of print.

pg
As a data point, in the source code of Arc and its libraries + the web server + News.YC + all of YC's internal applications, there are 738 calls to def (= CL defun) and 187 calls to mac (= CL defmacro). This isn't a complete list, because there are other defining operators, but it gives you an idea how large a role macros play in a typical Lisp application.

If you measured by uses of each type of operator, the role of macros would be even larger. You often write a function that you only call once, but you wouldn't do that with a macro.

Here's the last piece of code I wrote. I wrote it because I was curious to see what colors users had chosen for the top bar.

  (defopa topcolors req
    (minipage "Topcolors"
      (tab
        (each user (sort (compare > [karma _])
                         (keep [aand (uvar _ topcolor)
                                     (isnt it (hexrep orange))] 
                               (keys profs*)))
          (tr (td (link user (user-url user)))
              (tdcolor (hex>color (uvar user topcolor)) (hspace 30)))))))
There are 10 macros used here: defopa, minipage, tab, each, karma, aand, uvar, tr, td, tdcolor.

Actually it's a hack that karma is a macro. I wanted it to be settable and was too lazy to define a setter, so I just defined it as a macro. It should be a function, and I should be able to say just (compare > karma).

Tichy
I thought that Arc is written in Scheme, not in Common Lisp? Are Scheme Macros equally powerful?

Edit: OK, perhaps Arc is written in Scheme, but Arc itself features CL like macros, that would explain it ;-)

eru
As far as I know Arc is written on top of MzScheme. Besides its 'clean' macros - MzScheme supports CL like macros, too. But your explanation could be correct as well.
pg
Arc is written in Scheme in the sense that it compiles into Scheme, and uses the Scheme read, numbers, etc. But Arc macros are not Scheme macros.
apgwoz
Scheme also has "CL" like macros that are hygienic. Scheme 48 for instance, supports explicit renaming macros and MIT has syntactic closures. However, most schemes have a library that adds your traditional defmacro style macros, CL users are likely to want, but there use is almost always discouraged.
pg
I just entered this variant into the server's repl so everyone can see the range of choices:

  (defop topcolors req
    (minipage "Custom Colors"
      (tab 
        (each c (dedup (map downcase (trues [uvar _ topcolor] (keys profs*))))
          (tr (td c) (tdcolor (hex>color c) (hspace 30)))))))
Defopa means only admins can use something, but since this is defined with defop, everyone can see this page: http://news.ycombinator.com/topcolors.
sridharvembu
(before I begin: I am familiar with macros & DSLs, code generation, building a custom language layer and then write your app in that custom language and so on, and I buy into almost all of it)

I am sure the code above is poetry for those deep into it, but for a person who is a bit rusty (or god forbid, new to the code base), this is basically indecipherable. Look at how many tokens are there in a line like

(each c (dedup (map downcase (trues [uvar _ topcolor] (keys profs*))))

How many people can answer the question of which ones are macros, functions, simple identifiers and so on? In that particular example I am stumped at [uvar _ topcolor] and can't get past it.

Yes, I have read Paul's thesis on Lisp code density (quick summary: each line is dense, but there is a lot less overall code so it evens out or better), but my counter would be that there is a density beyond which most mortal brains can't penetrate.

Without intending to cause offense, I am going to say that this is tending to write-only code. Sure, the equivalent Ruby or Python (leave alone blubs like Java or C#) would be pages long, but I suspect that lower density leads to better understandability.

pg
Look at how many tokens are there in a line like

There's not really a concept of lines in Lisp; you can break expressions wherever you like. E.g. you could also write:

  (defop topcolors req
    (minipage "Custom Colors"    
      (tab 
        (each c (dedup (map downcase 
                            (trues [uvar _ topcolor]    
                                   (keys profs*))))
          (tr (td c)
              (tdcolor (hex>color c) 
                (hspace 30)))))))
sridharvembu
Actually, the equivalent Ruby/Python would use a simple SQL query. I have long noticed how SQL is the poor-man's functional "DSL" for such uses ...
nostrademons
I don't think that's a problem with macros so much as obscure naming conventions and an obsession with brevity. In the original example, I had a pretty good idea what minipage, tab, each, dedup, map, keys, downcase, tr, td, and tdcolor mean. I can figure out c (color?) and req (request?) from context. I have no clue about defopa, uvar, trues, _, [], profs, topcolor, hex>color or hspace (or I have guesses, but wouldn't trust my guesses enough to pin down the semantics).

From PG's post above, it seems like those are fairly evenly distributed across macros and functions. I don't need to know whether td is a macro or function to guess that it outputs a table data cell. I do need to know that it outputs a table data cell, and I only know that because I'm familiar with HTML.

sridharvembu
Agreed on the obsession with brevity.

Reminds me of mathematical proofs, where you work out everything offline, and then make the proof elegantly brief. There is a similar process going on here - writing macros really does boil down to that kind of offline work.

Just as with such elegant proofs, though, you do need a pencil and paper to trace through the steps. And if the programmer were anything but of the highest caliber, it is fairly easy to get tripped up.

The way blub languages handle (the absence of) macros for td, tr etc. is to invent templating languages (JSP etc) where you embed Java inside HTML. You basically have a DSL living in a separate syntactic space. That gets the job done, for the specific problem space at hand (web apps).

I have come to like that syntactic space separation: HTML lives as HTML, SQL lives as SQL and so on. With macros, you are getting everything to live in the same syntactic space, so you get the proliferation of tokens, without that structural separation that aids in comprehension.

shiro
I wonder what makes the difference in person's perception of code. I think I have a good idea of what the pg's code fragment does at a first grance. (I do know what [ _ ] means in Arc, though; pg mentioned it somewhere.)

I doubt it's the density that's the problem. I also doubt macros are. It's more like whether you recognize "the way" in which you're supposed to write, in the given language. Any language, verbose or terse, is indecipherable until you get familiar with the way even you know the syntax and semantics of the language.

sridharvembu
Granted. To be fair to Paul, I got about 60% of it based on a (rusty) idea of lisp. The _ got me (should've thought of Perl)... but now it is clear.

My broader point still holds though - there is a maximal density ...

And the reason I commented on it was that in posting that code fragment, the message was that it would be "kind of obvious" to any reasonably competent programmers how macros really help. I agree macros really do help, so no quarrel there.

In this case, we understand the "application" perfectly, and then we get to see the code. It still got (some of us) stumped, and (some of us again) actually know about macros and are moderately familiar with Lisp idioms. That is why I asked if density can go too far.

Compare that to something like LINQ, which I am coincidentally studying. It is a DSL for querying object collections (a similar use case to the code here). With no special background, you can go in, read code, and get a reasonable idea of what is going on.

LINQ actually lets you get rid of a lot of code (something I agree with in general), so it lives much higher on the density scale than a typical blub.

shiro
I see your point. So the question is actually whether a language designer can only care his "types"---those who have similar thinking habit and thus has less difficulty to understand the code---or should target at larger audiences.

Ultimately that's the choice of the designer; not all languages need to address to the broad range of programmers.

pchristensen
I don't think the point was that everyone was supposed to be able to understand every detail of the code. I think he was showing that by using a bunch of macros, he could add a new command to the web server that would create a new type of page with a very small amount of code. How would the code look to create the top color page if it was written in another language?

Less code = faster development, easier maintenance, fewer bugs

mnemonicsloth
You're just confused by some syntactic sugar. [uvar _ topcolor] is shorthand for (lambda (x) (uvar x topcolor)), according to "Arc at 3 Weeks"

each = mutant foreach

dedup = remove duplicates

(trues [] (keys profs*)) would mean "map the lambda (predicate) over the list of keys to the hash of profiles, returning a list of elements for which it returned #t."

A few more specifics about the application and language would make this comprehensible.

pg
Close, except trues returns the values returned by the function, not the elements for which the function returned true.

  (def keep (f seq) (rem [no (f _)] seq))

  (def trues (f seq) (rem nil (map f seq)))
tel
With dense code, you have to think about every word. With sparse code, you have to think along with every line/block/idiom. Both require the (same?) thought process, but with dense code you get two advantages:

- You can look ahead easier and get a bird's-eye-view - Once you learn an operator, recognizing it (and writing it correctly) becomes a single word instead of said line/block/idiom.

My guess is that [uvar _ topcolor] is a function similar to (lambda (x) (uvar x topcolor)). That's just a guess, but once I learn whatever it is, I'd wage that it's easier to recognize and debug the bracket syntax.

Zak
Let me see if I can translate that in to Ruby. Note that the expression isn't closed, so I'm changing it to:

  (each c (dedup (map downcase (trues [uvar _ topcolor] (keys profs*))))
    (do-stuff-with c))
which I think translates roughly to:

  keys(profs*).find_all{|prof| uvar(prof, topcolor)}.map(lambda {|str| str.downcase}).uniq.each do |c|
    do_stuff_with(c)
  end
This could be slightly off, as I'm having to guess at what some of this stuff does. It's almost as dense and no more readable.
[deleted]
(comment deleted)
paul
For me, your ruby version is more readable because the order is more sequential (like a pipeline). The arc version has to be read from the inside-out, vs left-to-right in ruby.
brlewis
I always read s-expressions left-to-right. The outer expression tells you what you're getting. I usually don't have to go very far in before I know what the code is intended to do.
icky
> and I should be able to say just (compare > karma).

I suggest you implement a terse syntax for partial-apply. For my own pet lisps, I use {}. (You are welcome to steal it ;-)

So {+ 1} would expand to (in scheme) something like (lambda rest (apply + `(1 ,@rest)). You could keep partial-applying arguments

  (do
    (= x {+ 1})
    (= y {x 2})
then actually apply it with ()

  (y 3 4 5) ; => 15
Your accumulator generator in Arc:

  (def foo (n) [++ n _])
Could be shortened to:

  (def foo (n) {++ n})   ; even terser than regular Arc!
The only time you'd need _ is when you want to change the order of the arguments (e.g. you'd never need {+ _ 2}, but you might need {/ _ 3}, or even {/ _ _ _ 3} if you want to jam the first three arguments in front.

This has some other interesting properties:

- In a lisp-1 OR lisp-2, #'foo can be represented as {foo}.

- Your karma macro can fit in there as (compare > {karma})

- With the _ variables, it might even be a general case of your [] syntax.

- {} stands out visually. "Here be electric magic."

- If you ever implement generic functions dispatched left-to-right, you can implement partial dispatch for partial application.

gregwebs
There is another facility ruby and lisp lack- a good typing system- which you can find in languages like Haskell.
sspencer
Some would call that a feature, not a bug.
sabat
Lack of strong typing in a language is usually on purpose (dynamic typing). But sometimes you want it. I like the idea of optional strong typing in a language.

Perl6, may it rest in peace, had that feature in its spec.

capablanca
Btw, will Arc ever be open-sourced?

I'd love to just sudo apt-get install arc in this lifetime.

pg
For once, I will answer this question: yes, soon. This winter, I hope.
tjr
Very exciting.

Should we arrange another Lightweight Languages day at MIT for this?

weel
More generally, somebody should arrange another Lightweight Languages day at MIT no matter what. Or maybe a Lightweight Languages days somewhere else. And wherever "they" organize it, it would be awesome if they could manage to webcast it and have people who cannot attend pose questions through IRC.

I attended Lightweight Languages once and it was awesome. Informal, a large variety of smart people, but each smart in their own way, that had a lot to genuinely learn from each other.

Sadly, conferences, even comparatively small and informal once, take some effort and some money to put together, and frankly it's not going to be me who is capable of coming up with either.

kirubakaran
We are waiting to type:

  git clone git://ycombinator.com/arc.1.0.0
Xichekolas
Just to continue this meme (YC's first?)...

I look forward to typing:

  emerge -av arc
kirubakaran
I meant:

  git clone git://archub.com/arc.1.0.0
stevecooperorg
Nah. We're waiting to download the windows msi file, which optionally installs a Visual Studio integration doodad and an eclipse plugin.

At least, that's what the people who want to become lisp programmers want. It's probably not what the current lisp programmers want. Hey ho. Just don't make emacs the default development environment.

kirubakaran
Emacs won't be default. pg uses vi. He highlights code segments and drags&drops them to repl.
stevecooperorg
Heh. Can't say that vi makes me feel much better ;)

Pg has a lot of people fired up about using lisps, but right now it's a sport you can only participate in if you've got a certain -other- set of skills like vi and emacs and a decent knowledge of system administration. Preconditions like this limit the user community massively. Small communities find it difficult to get momentum.

I've forced myself to stick with emacs/slime because basically that's that there is. I'm getting comfortable, and starting to like it. But I think it's actually a kind of Helsinki syndrome...

I'd like to write the next Viaweb, and I'd like to write it in Lisp. But if the basic language is inviting, the rest of the environment is sorta hostile; a shangri-la in a swamp.

So the quality of the installer, the libraries distributed with it, the documentation, and the bundled editor will ultimately affect it's success.

AcMC9wk0W5
I am interested foremost in reading a description or specification of Arc, not using an implementation.
tjr
Will the source to news.yc also be released, as a non-trivial example program?
lg
PG: Can an arc instance distribute its threads across multiple cpu's? I'm not asking "can-in-principle" but rather whether the current implementation has that feature.
Tichy
I stopped reading the Haskell tutorial when I came to the part where arrays have to be homogeneous (or something like that, basically all elements in the array have to have the same type, if I remember correctly). That is one of the things that is most annoying in Java, so why put up with it in another language?

In a similar vein, I wonder about the Erlang hype - some things are really impractical to do in Erlang, for lack of syntactic sugar and the immutable memory constraint. People were always bickering about the constraints in Java (no multiple inheritance and stuff like that), so why choose another language with impractical constraints instead?

jfoutz
This is a little tricky, Types can be pretty broad. First, you could define a tree type with all sorts of different leaves, you could have a list of tree parts, something like:

Tree = Branch Tree Tree | ILeaf Int | SLeaf String

then a list like [ILeaf 1, SLeaf "foo"]. Another way to go takes advantage of type erasure. Currently it's just a popular extension but it lets you have lists of stuff in the same type class.

A good example is "Show" you can have a list off stuff that knows how to turn itself into strings. always a handy thing.

Basically you get the big two, group things based on what they are, or group things based on what you can do to them.

Perhaps there is a big pile of random stuff you want to hang on to, but i'd bet 99.9% of the time you really want to collect the stuff into a coherent type.

enigman
In theory, it should be possible to get better performance out of a Lisp application, since most versions of Common Lisp include a compiler. Ruby is interpreted at present and can be slower in some contexts. Runtime pauses for garbage collection can be quite noticeable in Ruby.
damon
Ruby descended from lisp, according to matz. So in order to "see what you're missing" by using ruby, take a look at what he removed or changed from lisp. In particular he removed macros and created syntax. As others have commented, macros are a very powerful language feature.

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/...

brlewis
It's not that there's something you can do in Lisp that absolutely can't be done in Ruby. It's just that it's more convenient in Lisp, e.g. real closures instead of seven or eight closure-like constructs.

I took a look at that Closures in Ruby page you mentioned.

It says about Ruby: "This is quite a dizzing array of syntactic options, with subtle semantics differences that are not at all obvious, and riddled with minor special cases. It's like a big bear trap from programmers who expect the language to just work."

Zak
Ruby has real closures with semantics similar to Lisp: lambda expressions that reference variables from their enclosing lexical scope. It also has a number of constructs that create closures implicitly, and some others that may or may not be implemented that way, but you're not required to use them. You can use closures just like you would in Lisp.
jimbokun
Here's a good comparison:

http://markwatson.com/blog/2008/01/cool-common-lisp-elephant...

Performance is a big difference:

"Common Lisp compiled code runs much faster than Ruby: typical benchmark results are about 30 times faster - but Ruby's slowness is mitigated if a lot of processing is performed in native libraries like Ferret."

I don't think Ruby has all of the functionality of CLOS or the Meta Object Protocol. Here's Matz commenting on the lack of multi-methods in Ruby:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/...

"Most OO langauge includes Ruby see methods as "element of an object" This does not hold for CLOS. You loose the tight relationship but gain a new level of flexibility."

Beyond that, I would say pick a programming methodology and there is probably a CL implementation of it somewhere out there built with macros. That's due to the greater maturity of CL and "real macros" (which have been well covered by others already). And beyond macros, the "code as data" paradigm enabled by s-expressions has other uses (I think Steve Yegge mentioned "executable log files" as an example).

kyle-burton
Other features that have piqued my interest (besides the already mentioned, and vitally important, macros) are:

- conditoins and restarts - CLOS - reader macros (which are distinctly different from normal macros).

If you're talking about other Lisps (besides CL), like Scheme, it's worth looking at define-syntax and syntax-rules and how they deal with some of the intricacies of creating macros with define-macro.

mnemonicsloth
1. CLOS and the MOP have some very devoted advocates, and it's obvious that CL's object model contains a strict superset of the behaviors available in other languages. If you don't absolutely hate OOP, they're worth checking out [1].

2. Macros. Macros, macros, macros. Commonplaces like "code is data" and "program-writing programs" are useful characterizations, but they make it easy to underestimate how much of a productivity booster metaprogramming can be.

Despite regular practice and a lot of first-rate reading material, I stayed clueless about macrology until I got stuck with a big ugly infrastructure problem [2] that forced me to use a bottom-up style. At some point I macroexpanded my code out of curiosity and found myself staring at pages and pages of boilerplate. Some of it -- database code in particular -- even reminded me of stuff I previously would have done by hand.

Anyway, I guess I recommend the Kool Aid...

[1] OOP owes me my Freshman and Sophomore years.

[2] I was building a platform, but I can't talk about it.

vikram
From the Ruby side the difference is the quantity and quality of libraries that are available. You are likely to find stuff that is currently been worked on, with CL this can be a real problem. There are a lot of half done libraries, so you have to be willing to read a lot of code, either make improvements yourself or write some from scratch.
tx
All other Lisp-inspired languages, including Ruby, have added one important feature by introducing their biggest restriction: they went with more complex syntax abandoning Lisp's S-expressions. By doing so, they have created a substantial difference in how code and data are represented: whereas in Lisp everything, including your code, can be treated as data.

This is, in my humble opinion, the biggest difference between Lisp and others.

db4n
Macros can be used to automate function definition. You can't use blocks and HOFs for that, because definition occurs at compile time. You can also automate definition of other macros.

http://paste.lisp.org/display/54227

stefan_tilkov
I know that Ruby's metaprogramming facilities don't equal Lisp's, but does that matter in practice? i.e. is there a convincing number of metaprogramming use cases that can be implemented using Lisp macros, but can't be implemented using the Ruby meta object protocol?