HN Explorer

Show HN: Common Lisp running natively over WebAssembly for the first time (soi-disant.srht.site)

97 points by metagame ↗ HN
A month or so ago, I ported a Common Lisp implementation (npt) to WebAssembly to make a silly blog post, because I was bored and have a lot of time on my hands to waste with things like this (I don't have a job, and because I have next to no experience, these meaningless, silly projects tend to fill what time I do have).

This is significant as it's the first time Common Lisp in particular has ever been hosted on it; wasm has a few poor decisions in its design that make it less-than-conducive to being a target for Common Lisp, and a lot of the more interesting implementations require an implementation to already be on the platform for bootstrapping purposes.

My previous attempts using other implementations haven't gone so well, despite throwing a lot of time at it (as an example, I have a fork of Eclipse Common Lisp, a defunct implementation from the 1990s, sitting on my disk with a few hundred lines of changes that I finally got to successfully compile and run a handful of very basic expressions, but it blows up when you try and define anything). In comparison, I was pleasantly surprised by how little I had to do, even though I did end up scrapping loads of lines of my own changes to npt in the process as I got a handle on how to make it work acceptably.

The Emscripten toolchain and I don't get along, partially because I don't like inlining ECMAScript into my C and vice-versa, so it's little more than a neat little demo right now.

You can load slightly more complex programs into it by hijacking the "imp" ECMAScript function every few hundred milliseconds with strings containing complete forms (this is essentially a batch processor, so there's no interactivity that allows it to wait while you decide what the rest of a form should be). Only one at a time, though. It's not that fancy.

If you mess up at all, even just a little error, it will crash. This is by design; I disabled the debugger. It's a giant hack, and the hack I eventually decided on left it impossible to have a debugging experience, with the benefit of getting to use a closer-to-unmodified npt.

This could be more useful, if I spent more time on it, but it's more fun if it's just a demo. I hope you enjoy the toy I made for you.

https://en.wikipedia.org/wiki/Batch_processing

If you don't know what forms are in the context of Common Lisp:

http://www.lispworks.com/documentation/HyperSpec/Body/03_aba...

http://www.lispworks.com/documentation/HyperSpec/Body/26_glo...

71 comments

[ 4.5 ms ] story [ 149 ms ] thread
[deleted]
(comment deleted)
kotborealis
Great work! A small nitpick — the page is a bit hard to read, links have low contrast, you can check it using accessibility tools.
rsstack
I initially rolled my eyes that it's yet another pointless text contrast complaint - and then I opened the site. My eye sight is great and that contrast (and border) is not great :)
[deleted]
(comment deleted)
Existenceblinks
I hope someone would create a tutorial which is using a toy programming language to compile to webassembly from scratch. Using existing language is too opaque to understand anything.
[deleted]
(comment deleted)
metagame
That sounds like a good idea. Have you considered writing it?
Existenceblinks
I was trying to compile my internal data and encounter "what's the right way to represent string in Webassembly" so that's why I came up with the demand for tutorial from a toy language because it has less features to worry about.

I don't understand why I got downvoted, it seems HN users with power to downvote just see everything is a nail they wanted to exercise their power.

Anyway, I'd probably write one when I actually implement it. I know linear memory is available but not sure whether or not having GC or mechanism that make memory safe.

gumby
Great work! You write,

> …wasm has a few poor decisions in its design that make it less-than-conducive to being a target for Common Lisp…

Could you say a bit more about those design decisions?

mbrock
I think a major limitation is that a WebAssembly module can’t run dynamically generated code, which is a huge part of typical Common Lisp implementations.
pmoriarty
"a WebAssembly module can’t run dynamically generated code"

Why does it have that limitation, and is there any hope that it will someday be overcome?

aidenn0
On your computer, heap, stack, code, and global data all share the same address space. On the WASM virtual machine, only heap and global data share the same address space. Code and stack are abstracted away by the machine.

It was designed to support C-like languages, so you can do the equivalent of loading a DLL (or dynamic shared object). However in Lisp it is not unusual to compile just one function at a time. It's definitely possible to create a DLL for each function and load each DLL, but it requires a completely different architecture than most Lisp implementations use (in which, when you compile a function, it just writes the machine code to the heap).

There are similar impedance mismatches with so-called W^X systems (where you are disallowed from having a page be both (W)ritable and e(X)ecutable.

nickmain
You could certainly generate a new Wasm module on the fly and then execute it from Javascript. Linking and sharing of memory should be possible.

Pyodide can dynamically load libraries that are separate Wasm modules so it is worth checking out how that works.

aidenn0
IIRC inter-module calls are more expensive than intra-module calls though.

WASM also doesn't allow for functions with variadic return counts.

jhgb
I was thinking about this, and one solution that came to my mind was serializing the heap into a temporary image and restarting the environment from an updated Wasm module with new code. This would also collect all garbage at the point of transitioning to the new code. You might need to be very careful to do this reasonably quickly, but in principle I don't see why this wouldn't work. The idea came to me when I was thinking about how an interactive-but-native-speed Oberon environment could be implemented in Wasm.
miloignis
Not OP, but as a fellow Lisp-on-wasm developer one problem is that the current release doesn't support tail call elimination. There's a proposal for it (a tail_call instruction), and Chrome has implemented it, but the Firefox/Spidermonkey team hasn't prioritized it, so it's sat for a couple years. At least two implementations (maybe two browser implementations? I don't recall) are needed for standardization, so things remain.

You know, I wonder how seriously I could be taken if I duck-taped a wasm-capable browser together out of Servo and Wasmtime to make a second implementation and push it forwards...

gumby
There is no branch instruction at all?
miloignis
There is, but all control flow is structured and the argument/return address stack is out of your control. You have blocks, loops, and if/else statements. The behavior of a branch instruction differs based on context - in a block, it jumps out of the block, in a loop it jumps to the beginning of the loop.

If you really wanted full tail-call behavior, you would either have to compile every function into the same mega-function in a loop and have some sort of if/else tree, or use trampolines (which would additionally require either storing parameters in memory somewhere or using the same type signature for all functions, since function calls are typed).

Overall, it's not a super great situation for true tail-call elimination. For now, I've implemented limited tail-call elimination for single-function recursive calls (transforming them into an in-function loop), and that's patched things up enough for me to continue working for now until I either need to come up with an optimized trampoline or the tail_call instruction finally gets standardized.

metagame
Thanks for the only compliment in the thread! I appreciate it, especially given I've been such a fan of your writing and life for such a long time.

I think that the people who responded to you covered much of it, but you can find more by doing a web search for it. I'd find you links myself, but it's early in the morning and I'm a little tired.

moonchild
I believe there are challenges related to nonlocal transfer of control, as well as multiple return values. More damningly, if I recall correctly, lisp implementors were consulted and their feedback ignored (just like apl implementors with .net, back in the latter's infancy).
catchclose8919
...god, my eyes ...why is everything CL plagued by such horrible design choices (hyperspec, Lisp-IDEs... all!) - why such ugly colors, ugly typography, bad contrasts, ugly logos, ugly diagrams, ugly supporting graphics?!

I know that even the language itself is kind of the opposite of "beautiful", but the way all docs, blogs, websites etc. look ...seriously, is this intended to scare away any aesthetically sensitive people? Programming languages are about aesthetics too, and Lisp at its core (not CL ofc) is absolutely beautiful!

mpfundstein
i agree. its amazing. yhe hyperspec is unreadable anyway... never really helped me at all
pfdietz
I've spent many hours doing deep dives into the Hyperspec. I have no serious problem with it.
emacsperson
Its actually quite readable in eww, the emacs web browser :^)
christophilus
Oh, wow. I thought you were just another HN hater, but nope. I think my eyes are bleeding.
metagame
I took a picture of a box of pens, snatched the hex values from it, and then threw them directly onto the page. I'm glad you like it!
User23
I like the cut of your jib sir.
brokenkebab2
Talking about the page behind the link - it's not like Sistine chapel, of course, but it's completely ok. In digital realm your perception is very strongly affected by medium/device/environment. Maybe that's the case. Not commenting about hyperspec though
jedimastert
As someone that has (admittedly minor) struggles with reading and visual processing, I would call the accessibility at least absolutely atrocious.

I don't know if I feel more or less betrayed by the fact that the HTML seems to be more or less bespoke and a screen reader would have no problem with the page, but all of these choices feel all the more intentional.

metagame
The page was written to render well if you turn reader view on, and the color schemes are all in one line and easy enough to change. The only thing missing from the reader view is the aside, which is silly and in proper contrast on the original page.

However, the contrast is WCAG AA-conformant (except for the links that aren't in a black box, which aren't important links, as I went out of my way to confirm as I wrote the post). The page is actually pretty accessible.

Accessibility is important to me, as many people I've known in my life have been disabled, but so are silly aesthetic choices.

It was intentional, but it also was well-intentioned. That's why it's readable in more or less every web browser, regardless of whether it supports the one line of CSS I'm using, and accessible to screen readers (although I didn't throw in any elements specifically for them, the page is written simply enough that it should work, intentionally).

alpaca128
The contrast between light blue for links and vibrant yellow for the background is absolutely not okay, I find it almost impossible to read.

Yellow as background can work well, but this is not the way.

stareatgoats
I think aesthetics is sometimes equated to "superficiality", "commerciality" even "girlishness" and the surface impression tends to be the polar opposite for niches that really want to distance themselves from anything such. Which is not commendable but there you go.
jedimastert
Which is really interesting given how male-dominated the web design field is
stareatgoats
I assume that was a convoluted way of disagreeing with the attribution of (some) "girlishness" to aesthetics? Suffice to say, the male domination doesn't by itself exclude all (pleasing) aesthetics, except in those niche corners of the internet with the "no girls allowed" sign on. At least maybe, it's really just a theory ;-)
metagame
I don't think you hit the mark at all, as the person who wrote the page. Nothing you note is a bad thing; they're all positives. Superficiality is my specialty.

Your comment kind of shows some biases you might want to work on.

stareatgoats
I hardly think I'm wrong just because you don't (think you) fit the bill. I may still be wrong, but it would take more than one single indignant denial to prove.

But I grant that my comment was flippant and OT, potentially ad hominem denigrating even if unintended. I'll try better next time. You on the other hand might try harder to make websites that are readable to ordinary people. Or not.

metagame
I think it's fine to accuse people of things, even though I did feel a little sad about it. I don't think the problem is that you broke rules of a web forum, or that you attacked me. I wasn't complaining that you were flippant and off-topic, I was complaining that you think silly web designs are a masculine trait, which is incredibly sexist, as well-intentioned as the sentiment is. Geocities was almost gender-balanced, which included a lot of people of all different kinds of backgrounds making strange-looking web pages. Neocities has more web pages by women than nearly anywhere else on the Internet, and many of those designs aren't to corporate sensibilities, either.

My post is WCAG AA-compliant, except for the very specific occurrence of meaningless social links, and those meaningless social links render well in a browser's 'Reader View'. My site contains no ECMAScript aside from what was necessary to get the toy working. I care deeply about making things accessible; it's important to me.

stareatgoats
> you think silly web designs are a masculine trait

So, this seems to be a simple misunderstanding then. Because that's not what I mean (nor wrote). Anyhow, although we can probably continue to debate the finer points of this, I propose we don't. Have a good evening.

dtagames
So true! Also, there are no decent UI frameworks for Lisp, so it's impossible to build a full stack app that looks good in a modern browser without adding a TS or JS web component layer. And... There's no modern IDE for Lisp.

I think this aspect of ignoring UIs and aesthetics has seriously held back CL.

lobstrosity420
> Also, there are no decent UI frameworks for Lisp

How about CLOG and cl-cffi-gtk?

dtagames
Ok, thank you for those references I had not seen. CLOG looks like it creates the UI separately and essentially "streams" it to you over a websocket. While novel, that introduces it own problems. Perhaps this why there aren't any working examples that I could find?

The CL library you linked is for connection to GTK, a desktop UI scheme. So, while interesting, that doesn't really help someone who is trying to develop a browser-native app, which is where the world has gone at the moment.

I think Lisp got left behind on the journey, and I think this UI problem is one of the top reasons, the other one being terrible-non-IDE-like-substances.

lobstrosity420
The CLOG approach is not that novel. It's the same tech that Laravel Livewire, Rails Hotwire and Phoenix Liveview are pioneering. I mentioned GTK because your original comment hadn't mentioned you needed browser-native functionality specifically. For that, you would be well served with CLOG or you can try something like HTMX with a CL backend that serves just HTML.

I wouldn't go as far as to say CL got left behind in the journey, it's still very much alive. But if you want to go with something with more mainstream appeal (not a bad want IMO) there's always Clojure.

[deleted]
(comment deleted)
metagame
That's not true at all, and I'm tempted to write something to prove you wrong.

My aesthetic choices were intentional, and almost nothing in your comment is right.

smcn
I'd love read that rebuttal. And I'd ensure that I read it in its gloriously yellow form.
hexo
Still better than 3/4 of current web.
coryrc
The hyperspec is copyrighted and licensed in a way where we can't change it at all. It's like the primary source to learn so impossible for anyone knowledgeable to clean-room reverse engineer a new one.
catchclose8919
...you're not joking, right? jeeezus
fami-com
http://clhs.lisp.se/Front/Help.htm#Legal >Permission to copy, distribute, display, and transmit the Common Lisp HyperSpec is granted provided that copies are not made or distributed or displayed or transmitted for direct commercial advantage, that notice is given that copying, distribution, display, and/or transmission is by permission of LispWorks Ltd., and that any copy made is COMPLETE and UNMODIFIED. IN PARTICULAR, the material that MUST appear in the copy includes:

>...

>Permissions related to performance and to creation of derivative works are expressly NOT granted.

>Permission to make modified copies is expressly NOT granted.

>Permission to add or replace any links or any graphical images to any of these pages is expressly NOT granted.

It's one of the most restrictive licences.

jhgb
At least in my country you'd be perfectly within your rights to distribute a program that would modify it automatically for every reader who'd want to read a modified version, since as per our copyright act, you're entitled to do pretty much any modifications to the copyrighted works that you possess, as long as you don't redistribute them.
jhgb
> why such ugly colors, ugly typography, bad contrasts, ugly logos, ugly diagrams, ugly supporting graphics

What? It looks perfectly fine on my monochromatic Genera monitor!

Jokes aside, Firefox's reader view makes it only marginally more readable. I wonder if it took the author extra effort to make it look like that, and whether there's a better page sanifier than the reader view.

metagame
All of the CSS is handwritten for exactly the intended effect that it gave.
[deleted]
(comment deleted)
FraaJad
So much complaint about color schemes. Hitting the Reader View formatted the page beautifully.

I'm thankful it is a simple HTML page that could be easily formatted using browser-built-in tools.

ngcc_hk
Is there something like a box that you can key in something and it works. Reader view give you text. Also that box?

I lost … perhaps I expect a REPL or … anyway not good to sell CL-webassembly I think.

metagame
As it says in the article, stdout is wired to the browser console. So you open your browser console to see it.

It's not meant to be sold, it's just a little toy.

[deleted]
(comment deleted)
ngcc_hk
Like reading Rfc in 1990s … a bit odd choice to use this format and font to sell anything these days.
[deleted]
(comment deleted)
metagame
Nothing's being sold; it's just a little toy.
aidenn0
This probably doesn't count as "natively" but I've run ABCL[1] under Doppio[2]. Startup times are under a minute in Chromium based browsers and under an hour in Firefox. I've run into zero stability issues, but its no speed demon.

[edit]

Just tried again today and Firefox gets to a REPL in about 3.5 minutes, while chromium is still right about 1 minute.

1: https://abcl.org/

2: https://plasma-umass.org/doppio-demo/

metagame
Thank you for making it!

Unfortunately, a JVM on wasm would be quite difficult for the same reason that Lisp over wasm is quite difficult (I had actually looked for wasm JVMs before trying anything). I had no idea there was a JS JVM implementation! That's very cool.

kaba0
There is also TeaVM which is very fast and has both a js and a wasm target.
aidenn0
TeaVM lacks reflection and a class loader, both of which are necessary for ABCL.