Congratulations!
I once bought your eBook on Hy, and still today I regularly receive notifications about your book having been updated. Thank you for your steady contributions. I really want to use Hy in one my production apps one day.
I enjoyed the less serious part a lot. I wish more programming related projects could embrace the whimsical. That might the best way to honor the python tradition in any case :)
I eliminated a lot of whimsy from Hy and its documentation years ago because it was distracting and created noisy test failures, but I did go too far at some point, and have tried to reintroduce a little whimsy more recently.
Well, this is a little embarrassing: Clojure was one of the biggest influences on Hy in its youth, but that was mostly before I got involved in 2016. I never actually learned Clojure. So hopefully somebody who knows both Hy and Clojure well can answer. I can tell you that at run-time, Hy is essentially Python code, so Hy is more tightly coupled to Python than Clojure is to Java; a better analogy is CoffeeScript's relationship with JavaScript.
I get the impression that Clojure tries to convince the programmer to avoid side-effects a lot more strenuously than Hy does, but it's still not a purely functional language, so I don't know how consequential that is in practice.
Clojure has a good collection library with immutable/persistent data structures, but as a language it allows side effects and has some mechanisms to manage them. It is also possible to call any Java method from Clojure.
Clojure does not work with Java ASTs, it translates into JVM bytecode directly.
I haven't used Hy, but I am the maintainer of a Basilisp which also compiles to Python and aims for reasonably close compatibility with Clojure if you're interested.
Wondering how custom immutable data structures fit in with the Python ecosystem.
Particularly, I know that NumPy arrays and Pandas Series/DataFrames are the popular data structures used in research computing in Python (for Statistics, Data Science, Machine Learning etc.). These data structures afaik are mutable, however (for performance reasons), so at least the aspect of immutability from Clojure cannot be easily integrated with the Python ecosystem.
This project is much younger and used by many fewer people than Hy, so I couldn't really speak to this besides my own opinions. The few who have started using it and contributing seem to just be using it as a way to write Clojure while interacting with popular Python libraries and tools. Kind of the same way that interacting with the Java ecosystem is often more pleasant from Clojure (IMO) than in Java itself.
I've tried to facilitate strong Python interoperability despite the variety of otherwise incompatible features of each language. It's trivial to work with immutable data structures using Clojure idioms and then convert them to Python data structures (as needed) at the boundaries, but the immutable data structures used by Basilisp are also generally compatible with Python's core (read-only) interfaces so that conversion may also not be necessary if you aren't expecting the called function to perform any mutations.
I'm not sure. I was going to say that Mojo is proprietary software and so I've never tried it, but I just checked and apparently it's free now. If nothing else, you can probably get a lot of Hy code to run on Mojo via `hy2py`, if Mojo supports a lot of Python as it claims to.
Edit: actually, confusingly, the GitHub repository for Mojo doesn't have an interpreter. The language is still proprietary.
Congratulations -- and thank you! I've been playing with Hy on and off (tried to do transformers with it, and then released https://github.com/kunalb/orphism written in hy). Time to pick it up again and take it for a spin
Wow! It has come such a long way since its early, humble beginnings.
I saw the original lightning talk that introduced Hy to the world at Pycon those ages ago. Soon after I met Paul and started contributing to the early versions of Hy. I was responsible for the CL-style kwargs (you’re welcome), some minor innards, and a library or two.
Whimsy is useful, especially to keep enthusiasm up. It’s nice when hackers can be hackers and not every thing is business.
While I haven’t been involved in years it brings a smile to me face to see the project continues apace. What a great milestone!
1. Does it support REPL-driven development? (condition system, breakloop, etc.)
2. Is there a standalone distribution? Distributing python in itself is a hassle, ideal situation would be to simply distribute a single Hy binary that contains all dependencies within it (either statically linked or as a zip file extracted in tmp directory).
Except uv doesn’t support conda so there goes many of the niche scientific packages required for many users like me. Someone please prove me wrong because I do love uv when I can use it. I’ve found pixi to be an ok alternative but not nearly as fast.
1. I don't know what a breakloop is. Hy uses Python's exception system, which is more like a traditional exception system than Common Lisp's condition system.
A breakloop is a REPL operating in the context of condition handling. When a condition is signaled, you can use the breakloop to modify state and direct how the condition should be handled (including fixing something local and letting the current function proceed by ignoring the condition).
Seems like that would only be doable by altering CPython to at least have a hook in the initial exception processing (or maybe there is some magic double-underscore thing for that already?).
I see. That's pretty similar to the feature set of [pdb](https://docs.python.org/3/library/pdb.html). You may then logically ask "Does Hy support pdb?". The answer is "sort of". I've fixed one or two bugs, but we don't test it. I suspect there are various features of pdb that assume Python syntax and would need some hooks to get working properly with Hy.
1. It supports the same set of features that python supports, which is pretty good when it comes to things like traditional step through and postmortem debugging. And CPython Supports a lot of internal hooks if you want to do really advanced dark magic. But it doesn't have anything like the condition system or handlers/restarts.
I was having some difficulty figuring out how Hy actually is translated to Python (and wasn't even sure if it was compiled or interpreted). Eventually I found on Wikipedia the following:
> Hy is a dialect of the Lisp programming language designed to interact with Python by translating s-expressions into Python's abstract syntax tree (AST).
I kind of wish this was made more clear on the main website. Perhaps, instead of introducing Hy as "a Lisp dialect that's embedded in Python", introduce it as "a Lisp dialect that compiles to Python's AST". The words "embedded in Python" don't make it very clear just how it's embedded into Python. The various ways you can embed a Lisp look very different and have very different tradeoffs.
For example, off the top of my head, I could "embed" a Lisp by writing an interpreter (in C if I care about performance) and letting it be called from Python, perhaps passing in a Python list instead of a string to make it more "native". Or I could "embed" a Lisp by compiling to Python bytecode. Or I could "embed" a Lisp by translating it directly to Python source code. Etc.
The "embed" part stems from the fact that you can mix Python and Hy in a project with bi-directional calling. Works great, because it is all Python byte code in the end.
The original hy annoucement makes it clear that they embed a Lisp by compiling with Python bytecode. You can see it in the following video about the 16:25 mark
> The various ways you can embed a Lisp look very different and have very different tradeoffs.
Hy itself provides options. Typically the process is that the Hy source code becomes Python AST objects, which Python then complies and executes, but you can also translate the Python AST objects into Python source text. Or you can use Python from Hy or vice versa: https://hylang.org/hy/doc/v1.0.0/interop
> Hy is a Lisp dialect that's embedded in Python. Since Hy transforms its Lisp code into Python abstract syntax tree (AST) objects, you have the whole beautiful world of Python at your fingertips, in Lisp form.
At long last! Now I can finally clean up https://github.com/rcarmo/sushy (I've been poking at it over the years, but every time I upgraded hy portions of the syntax broke, or things would get moved in and out of the hyrule package, etc.)
By the way, Hy works really well inside https://holzschu.github.io/a-Shell_iOS on the iPad, although the syntax highlighting in vim/neovim needs to catch up to the 0.29+ releases and async.
Although I've tried using Fennel and Guile instead over the years, having access to Python libraries and ecosystem is preferable to me, and with async I can do some very nice, efficient API wrangling (doing HTTPS with fine-grained control over socket re-use and headers remains a pain in various Schemes, so I very much prefer using aiohttp)
> Are there Python language features I can't use in Hy?
At the semantic level, no. I work to cover 100% of Python AST node types with Hy's core macros. It does take me a little bit to implement a new core macro after the CPython guys implement a new feature, but you can always use the `py` or `pys` macros to embed the Python you need, should it come to that.
> Or performance penalties in using Hy?
Compiling Hy (that is, translating it to Python AST) can be slow for large programs (I've seen it top out at about 3 seconds), but at runtime you shouldn't see a difference. Hy always produces bytecode, which can be used to skip the compilation step if the code is unchanged.
You take a little performance hit upon initial startup (from a clean filesystem, while __pycache__ folders are created). Other than that, mostly everything is the same.
I'm now figuring out how to pack images to OpenAI REST calls (using my own REST wrapper), and everything is peachy. Here's my test snippet (mostly to b64encode the file):
Lack of self-contained tooling. Idle doesn't work with Hy. You'll probably need to fiddle with Emacs to set your environment first, before being able to do anything beyond playing with the language in the REPL.
If I were to guess, it's to be able to use the all the packages in the Python ecosystem, directly. It's for situations in which Python is already a given. In fact, it's probably the case that many Python programmers can't even use this, due to being in a situation in which even the poor syntax is nonnegotiable.
Years ago, under the influence of Lisp romanticism late into my university years, I worked on a domain-specific language for designing and analyzing control systems as my senior design project, using Hy! Just checked, it's been five and a half years to be specific. Really, time flies.
I looked the examples page, but it was a little disappointing. Every example was something that was easier (and sometimes shorter) in Python.
It would be awesome if there were an example of something that can't be done in Python because it takes advantage of lisp's "functions are first class".
Yay! The birth of a language is a beautiful thing.
I’m curious about the macros: how are these implemented? They seem like pretty straightforward unhygienic Lisp macros, which is a little bit of a disappointment, but better some macros than none at all! Anything about the macro system that distinguishes it from the Common Lisp system? E.g. anything borrowed from Scheme or Racket? Docs are sparse here.
It’s far from new. In 2012 I worked for a shop who used an internal package named “hy”, and the introduction of this Hy made our builds break in a novel and interesting way.
(Also, use something to insure your own internal packages have a higher priority, alright? That’s a lesson I didn’t need to learn twice.)
Yes, and it's a very nice tutorial! I'm interested in implementation details. Maybe there's no hygiene (and no scope sets etc.) to worry about—that would probably make documentation a little shorter. I'm sure the documentation will grow as people run into edge cases.
(I'm also probably a little spoiled with documentation coming from Racket which has like 4 big chapters dedicated to different aspects of macros scattered around the docs, plus some associated papers. Forgive me—I'm not trying to dunk on Hy; I just like reading docs.)
Admittedly, I've tried not to document the implementation. Yeah, they're pretty much simple dirty Common Lisp macros. Internally, they're functions that are called with the arguments converted to models (via `hy.as-model`), and then the return value is converted to a model. If a macro's first parameter is named `_hy_compiler`, it gets access to the current compiler object; this is undocumented since it's only meant for internal use. Reader macros have no parameters, but can access the current reader object as `&reader`. When it's defined, a reader macro is added to the current reader's dispatch table.
I have a hard time trusting the CL macros I write because of unexpected interactions with the context that I use them in. While it is the case that CL macros are more powerful than the R6RS macros-by-example system, Racket’s system (and some other newer languages that have adopted things pioneered by Scheme and Racket, such as Elixir) give you hygienic macros without sacrificing expressive power.
I want my macros to be easy to write correctly. That can only happen when the system has proper hygiene.
Table saws were only improved by the addition of an emergency stop to prevent people from maiming themselves. Power tools don’t have to be dangerous.
If you are wanting to introduce variable capture you better be really explicit about when you want it.
If there’s no hygiene, I have to know everything about how the macro is implemented in order to trust it and use it confidently. Might be fine for small shorthand, but that won’t scale. You need non-leaky abstractions to build on them.
Racket’s `syntax-parse` and “syntax parameters” show that you can have it both ways: procedural macros that are hygienic by default, but with an explicit escape hatch when you do want to introduce new bindings into the macro call site. It also gives you much much better errors.
CL macros are about as dangerous as malloc/free, but without years of experience and tools like Valgrind to debug. They’re hard to trust and get right.
Racket macros are like GC/affine typing: everything is correct by construction.
I've used table saws, with and without protection. They're all dangerous as fck, because removing the chance of getting hurt means converting it to a completely different kind of tool. Chain saws, same thing. Power tools.
Of course we want them to be as safe as possible, but that's a different discussion. All attempts I've seen so far have dropped functionality to get there.
> All attempts I've seen so far have dropped functionality to get there.
Well, then I recommend you take a look at Racket's macro system: Racket gives you hygienic macros without any loss of power. (It's actually more powerful and expressive than CL macros.)
The way you phrased that suggests you're only familiar with CL-style macros, where arguments to macros are nested lists of symbols a function or variable is known by it's name (a symbol) and nothing more.
Racket's model is much more sophisticated and powerful. The input to a macro in Racket is a syntax object [1], which combines the CL-like quoted expression with additional source and lexical binding information. This means that in Racket, unlike CL, a variable is not just it's name—it's also all this other information. Racket uses scope sets to track binding information in a sane and hygienic manner across different macros and functions.
So, if you want to introduce an identifier that the macro caller can interact with (note: I said an identifier—you can introduce any symbols you want but they'll be different identifiers because their scope sets will be different) you need to explicitly state that you would like to create an identifier with a particular scope set. [4]
But that's the old, dumpy, clunky way of doing things. Thanks to recent research, we have much better ways of introducing identifiers in a sane, hygienic way. Gregg Hendershott's excellent "Fear of Macros" walks through making the `aif` macro using syntax parameters [3] which let you cleanly introduce new bindings. (See the paper "Keeping it Clean with Syntax Parameters" he's linked to in his post.)
So, in short, no, we're not back to regular CL macros because Racket prevents us from accidental variable capture but gives us an easy way to do 99.999% of the use cases for breaking hygiene (syntax parameters) and then one more way (`datum->syntax`) just in case we really need to do something out of the ordinary. In either way, Racket lets you express your intent with macros better and more precisely than CL.
130 comments
[ 4.8 ms ] story [ 171 ms ] threadI just read through the author list on the Hy repo and had a glimpse into their blog posts. Cool stuff, great work.
(Does `let` work? I remember that being a barrier for a while.)
Could you compare the language with Clojure?
I get the impression that Clojure tries to convince the programmer to avoid side-effects a lot more strenuously than Hy does, but it's still not a purely functional language, so I don't know how consequential that is in practice.
Clojure does not work with Java ASTs, it translates into JVM bytecode directly.
https://github.com/basilisp-lang/basilisp
Wondering how custom immutable data structures fit in with the Python ecosystem.
Particularly, I know that NumPy arrays and Pandas Series/DataFrames are the popular data structures used in research computing in Python (for Statistics, Data Science, Machine Learning etc.). These data structures afaik are mutable, however (for performance reasons), so at least the aspect of immutability from Clojure cannot be easily integrated with the Python ecosystem.
I've tried to facilitate strong Python interoperability despite the variety of otherwise incompatible features of each language. It's trivial to work with immutable data structures using Clojure idioms and then convert them to Python data structures (as needed) at the boundaries, but the immutable data structures used by Basilisp are also generally compatible with Python's core (read-only) interfaces so that conversion may also not be necessary if you aren't expecting the called function to perform any mutations.
Edit: actually, confusingly, the GitHub repository for Mojo doesn't have an interpreter. The language is still proprietary.
I saw the original lightning talk that introduced Hy to the world at Pycon those ages ago. Soon after I met Paul and started contributing to the early versions of Hy. I was responsible for the CL-style kwargs (you’re welcome), some minor innards, and a library or two.
Whimsy is useful, especially to keep enthusiasm up. It’s nice when hackers can be hackers and not every thing is business.
While I haven’t been involved in years it brings a smile to me face to see the project continues apace. What a great milestone!
1. Does it support REPL-driven development? (condition system, breakloop, etc.)
2. Is there a standalone distribution? Distributing python in itself is a hassle, ideal situation would be to simply distribute a single Hy binary that contains all dependencies within it (either statically linked or as a zip file extracted in tmp directory).
https://docs.astral.sh/uv/guides/tools/#running-tools
(context: uv can install and manage python versions)
2. No, sorry.
Seems like that would only be doable by altering CPython to at least have a hook in the initial exception processing (or maybe there is some magic double-underscore thing for that already?).
>A convenient way to use this class to interactively debug code is to insert the following in the code you want to debug:
>Or in Python: >Note that as with `code.interact()`, changes to local variables inside the REPL are not propagated back to the original scope.Also, looking at the code on Github suggests this compiler is written in Python (see https://github.com/hylang/hy/blob/master/hy/compiler.py).
I kind of wish this was made more clear on the main website. Perhaps, instead of introducing Hy as "a Lisp dialect that's embedded in Python", introduce it as "a Lisp dialect that compiles to Python's AST". The words "embedded in Python" don't make it very clear just how it's embedded into Python. The various ways you can embed a Lisp look very different and have very different tradeoffs.
For example, off the top of my head, I could "embed" a Lisp by writing an interpreter (in C if I care about performance) and letting it be called from Python, perhaps passing in a Python list instead of a string to make it more "native". Or I could "embed" a Lisp by compiling to Python bytecode. Or I could "embed" a Lisp by translating it directly to Python source code. Etc.
Regardless, interesting project!
https://m.youtube.com/watch?v=1vui-LupKJI
> ...because this is a frontend like LLVM or GCC that compiles instead of bytecode, uh, to Python AST, um, so this Lisp compiles entirely to Python
@ https://youtu.be/1vui-LupKJI?t=1020
Yes, that's right. Hy is not self-hosted.
> The various ways you can embed a Lisp look very different and have very different tradeoffs.
Hy itself provides options. Typically the process is that the Hy source code becomes Python AST objects, which Python then complies and executes, but you can also translate the Python AST objects into Python source text. Or you can use Python from Hy or vice versa: https://hylang.org/hy/doc/v1.0.0/interop
> Hy is a Lisp dialect that's embedded in Python. Since Hy transforms its Lisp code into Python abstract syntax tree (AST) objects, you have the whole beautiful world of Python at your fingertips, in Lisp form.
By the way, Hy works really well inside https://holzschu.github.io/a-Shell_iOS on the iPad, although the syntax highlighting in vim/neovim needs to catch up to the 0.29+ releases and async.
Although I've tried using Fennel and Guile instead over the years, having access to Python libraries and ecosystem is preferable to me, and with async I can do some very nice, efficient API wrangling (doing HTTPS with fine-grained control over socket re-use and headers remains a pain in various Schemes, so I very much prefer using aiohttp)
More concrete: Are there Python language features I can't use in Hy? Or performance penalties in using Hy?
At the semantic level, no. I work to cover 100% of Python AST node types with Hy's core macros. It does take me a little bit to implement a new core macro after the CPython guys implement a new feature, but you can always use the `py` or `pys` macros to embed the Python you need, should it come to that.
> Or performance penalties in using Hy?
Compiling Hy (that is, translating it to Python AST) can be slow for large programs (I've seen it top out at about 3 seconds), but at runtime you shouldn't see a difference. Hy always produces bytecode, which can be used to skip the compilation step if the code is unchanged.
I'm now figuring out how to pack images to OpenAI REST calls (using my own REST wrapper), and everything is peachy. Here's my test snippet (mostly to b64encode the file):
This shows you async, context managers, selective imports, f-strings... etc. All that you need, really.Why not just use something closer to the metal: Common Lisp, Scheme, Clojure, Racket? Especially, use a compiled language, instead of an interpreter.
https://github.com/digikar99/py4cl2-cffi
https://github.com/marcoheisig/lang
[0] https://github.com/hylang/hy/discussions/2609
Hehe, clever.
Years ago, under the influence of Lisp romanticism late into my university years, I worked on a domain-specific language for designing and analyzing control systems as my senior design project, using Hy! Just checked, it's been five and a half years to be specific. Really, time flies.
Here it is for anyone curious: https://github.com/celaleddin/gently
Since then, I've been following Hy from a distance and it's amazing to see it's still active. Thank you everyone involved!
It would be awesome if there were an example of something that can't be done in Python because it takes advantage of lisp's "functions are first class".
A Clojure-compatible(-ish) Lisp dialect targeting Python 3.8+ https://github.com/basilisp-lang/basilisp
https://www.codesections.com/blog/raku-lisp-impression/
[thanks to Larry Wall’s penchant for collecting stuff]
letrec, lambda, or & and are not functions in Scheme.
I’m curious about the macros: how are these implemented? They seem like pretty straightforward unhygienic Lisp macros, which is a little bit of a disappointment, but better some macros than none at all! Anything about the macro system that distinguishes it from the Common Lisp system? E.g. anything borrowed from Scheme or Racket? Docs are sparse here.
(Also, use something to insure your own internal packages have a higher priority, alright? That’s a lesson I didn’t need to learn twice.)
(I'm also probably a little spoiled with documentation coming from Racket which has like 4 big chapters dedicated to different aspects of macros scattered around the docs, plus some associated papers. Forgive me—I'm not trying to dunk on Hy; I just like reading docs.)
Quite the opposite, they are more powerful than the alternatives.
Macros are power tools, dumbing them down for safety is missing the point.
I want my macros to be easy to write correctly. That can only happen when the system has proper hygiene.
If you put your own symbols into generated code you better have a damn good idea what you're doing.
It's a power tool.
If you are wanting to introduce variable capture you better be really explicit about when you want it.
If there’s no hygiene, I have to know everything about how the macro is implemented in order to trust it and use it confidently. Might be fine for small shorthand, but that won’t scale. You need non-leaky abstractions to build on them.
Racket’s `syntax-parse` and “syntax parameters” show that you can have it both ways: procedural macros that are hygienic by default, but with an explicit escape hatch when you do want to introduce new bindings into the macro call site. It also gives you much much better errors.
CL macros are about as dangerous as malloc/free, but without years of experience and tools like Valgrind to debug. They’re hard to trust and get right.
Racket macros are like GC/affine typing: everything is correct by construction.
Of course we want them to be as safe as possible, but that's a different discussion. All attempts I've seen so far have dropped functionality to get there.
Well, then I recommend you take a look at Racket's macro system: Racket gives you hygienic macros without any loss of power. (It's actually more powerful and expressive than CL macros.)
Racket's model is much more sophisticated and powerful. The input to a macro in Racket is a syntax object [1], which combines the CL-like quoted expression with additional source and lexical binding information. This means that in Racket, unlike CL, a variable is not just it's name—it's also all this other information. Racket uses scope sets to track binding information in a sane and hygienic manner across different macros and functions.
So, if you want to introduce an identifier that the macro caller can interact with (note: I said an identifier—you can introduce any symbols you want but they'll be different identifiers because their scope sets will be different) you need to explicitly state that you would like to create an identifier with a particular scope set. [4]
But that's the old, dumpy, clunky way of doing things. Thanks to recent research, we have much better ways of introducing identifiers in a sane, hygienic way. Gregg Hendershott's excellent "Fear of Macros" walks through making the `aif` macro using syntax parameters [3] which let you cleanly introduce new bindings. (See the paper "Keeping it Clean with Syntax Parameters" he's linked to in his post.)
So, in short, no, we're not back to regular CL macros because Racket prevents us from accidental variable capture but gives us an easy way to do 99.999% of the use cases for breaking hygiene (syntax parameters) and then one more way (`datum->syntax`) just in case we really need to do something out of the ordinary. In either way, Racket lets you express your intent with macros better and more precisely than CL.
[1]: Racket syntax objects: https://docs.racket-lang.org/guide/stx-obj.html
[2]: Syntax model, scope sets: https://docs.racket-lang.org/reference/syntax-model.html
[3]: "Fear of Macros", writing the `aif` macro: https://www.greghendershott.com/fear-of-macros/Syntax_parame...
[4]: Reddit thread on making an unhygienic macro: https://www.reddit.com/r/Racket/comments/tpgaa4/is_there_a_w...