The problem comes mostly from lisp's dislike for text (my personal opinion). Lisps work best when everything is a list, always. As soon as you cross the boundary into... Well.. Anything else , really, you need a little bit of ugly glue.
That's why it's so ugly, in case anyone was about to make the "ermagerd if lisp is so great explain this..." comment. :)
[edit: which, btw, is not to lisp's credit, imho.]
I completely agree. This is actually better than a lot of cases because Java import statements are just lisp symbols delimited by periods. This is mostly only useful if you're transforming lisp statements into code. Transforming java to java or c to c is just a mess. Still, If I need to do some ugly code transformation, I'd rather it be in scheme than c. :)
Also, the example could and probably should be rewritten more cleanly. Hence the comment. That was the first thing I used the tool for and my scheme has improved since.
I don't get what you're suggesting. Add a hash type? The syntax isn't much different from a list, and sometimes it is a macro on top of existing lists.
In other words, what is a concrete example of list syntax being inadequate? Literally every data format of which I can think can be expressed as a list or a derivative of lists (e.g. association lists).
Yeah, except IntelliJ Idea is a refined product that does that and 100 other things.
I don't know what IntelliJ has inside of it, but I know Eclipse is based on EJC, the Eclipse Java Compiler, which can turn Java code into a Java abstract source tree, and then turn that either back to Java code or to byte code.
It's not too different from the read function in LISP but it is a bit more complex...
You did a great job and I am sorry that the discussion is mostly about the example. I saw the Scheme version and found it quite verbose, like some other people on this thread. That's what made me want to implement a CL one. Maybe if you had a less controversial example where you produce e.g. Markdown text, it would not distract people from the tool.
It was verbose. I actually left a comment in the example about how it could have been written better. Someone else further down in the comments suggested a nicer way of writing it. I think most people got the idea though! Thanks for the feedback. :)
I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE.
That being said I use code generation instead of the IDE all the time but for harder things like generating fluent builders , copying objects, generating visitor pattern for pattern matching etc.
What I do is write Groovy scripts and will either use Groovy GString templates or Squares Poet to generate code.
What I do though instead of making some sort of DSL is use a subset of the language itself to help the code generator. For example copying a list of private fields I can quickly turn into either an immutable object with builder or an interface or just a plain object copy etc etc.
I put my Groovy scripts in my ~/bin and then just keep my terminal running. I then copy from Eclipse/Intellij some code and then run:
Then paste back the code into the IDE. Ideally I would use APT libraries (and in same cases I have converted Groovy code into APT libraries) but some times I just want something lightweight.
> I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE.
Agreed that the example is weak. I do agree that, for import statements, having and IDE fill those in for you is fair game. It can figure out what you are trying to use and offer to import those, that's fine.
The problem here is the slippery slope. You get an IDE writing import statements. Than it will write getters and setters for you – which is already an aberration that prevents language evolution. For instance, C# has a syntax for default getters and setters (and for properties, in general).
Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet).
It's amazing the amount of machinery that gets created when people do not have access to a useful macro system.
I probably should have presented more than one example in my post. A lot of people are focusing on the example I presented and not the general utility. Live and learn I guess. :/
I like the effort and general idea/content but for me almost any sort of plain text code generation is just not that impressive.
It would have been interesting if you had for example generated byte code or enhanced some existing code (Java classes) but I realize that was is probably out of scope for the article. For example it would be funny/interesting if you had Guile VM generating JVM byte code in realtime and some how the Guile VM could figure out some sort of optimizations based on macros to generate better bytecode.
Yes but there is also abuse and or complexity that doesn't make it easy. One of the classic examples for me at least is OCaml's Camlp. For Lisp like languages it is easier to have a macro language but for typed languages I think it becomes far more difficult both to implement and understand.
A lot of people don't realize that Java offers a macro/preprocess like system (APT) and it is even sort of hygienic (an example project is Google's Auto [1]).
APT is not that powerful but it gets you quite far and going back to OCaml the new ppx language extension is actually fairly similar (ie annotation processing instead of a complete macro/preprocessing language). .NET has something analogous but even more powerful (IIRC as well as the whole access to the AST aka LINQ style).
> Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet).
2001 called and wants it example back :) . Yes APT didn't exist for a while. Tangental but XDoclet sort of reminds me of Go's very weak annotation system. And again It takes time to get a macro/preprocessing engine right particularly for typed
languages. e.g. Rust has had some fair challenges getting this right and even their macro system is not completely hygenic IIRC.
> ... but for typed languages I think it becomes far more difficult both to implement and understand.
I don't think that types are the issue but rather that such languages were not designed with macro systems in mind from the very beginning. Some added complexity is unavoidable, of course.
> A lot of people don't realize that Java offers a macro/preprocess like system (APT) and it is even sort of hygienic (an example project is Google's Auto [1]).
I didn't. Will look it up.
> 2001 called and wants it example back
I had to use that abomination as far as 2005. I understand that's a dated example, but it is the best example of that slippery slope taken to its extremes that I could think of. It is also a good example in the sense that it used XML instead of expressing itself in the original language. As such, it is also a nice illustration of Greenspun's Tenth Rule.
Calling it a macro system is probably a stretch of the word :) but if you check out APT you should also check out the far sexier Java Agents [1] which is code that changes other code during runtime (ie alters structure and provides meta programming).
> I don't think that types are the issue but rather that such languages were not designed with macro systems in mind from the very beginning. Some added complexity is unavoidable, of course.
I thought I had a reference somewhere to back that up but can't find it. However I do massively agree having had better planning early in the language could have prevented some serious headaches (its one of the reason I have concerns about Go).
That being said there are some that think macros are inherently code smell particularly if your language has lazy evaluation. e.g. Haskell programmers can go quite far with out having macros. That is controlling evaluation may free the need of a majority of macros (ignoring the verbosity saving aspects).
I really like the idea of a general-purpose text-replacement pre-processor like this. I especially like:
- A "real" programming language (Scheme)
- Only 1 character to avoid/escape (~)
- Expression-oriented (as opposed to, for example, impure echo/print)
- Completely orthogonal/independent of the code being manipulated (unlike the myriad Java/JVM examples cited by other comments)
One thing that would be nice is to distinguish between pure expressions and impure ones. That way, pure expressions can always be processed automatically (e.g. in wrapper scripts around tools like static analysers) since they're incapable of performing any effects.
Impure expressions would still be useful, e.g. to scan filenames in the current directory, but can be identified as such and skipped by tools/scripts.
Hmm, I think it's a dangerous thing to be skipping portions of code in analysis just because they're impure. I think you'd be better off just taking the code produced after processing and running that through your static analyzer as is. That's what's going to be running in the end anyway. Maybe I'm not fully understanding the problem though.
Well, you could choose to abort rather than skip anything.
The problem is that impure code can do dangerous things (depending on which i/o primitives are provided). For example, a macro might create and delete a bunch of temporary files; this might work fine in one scenario (e.g. during compilation) but misbehave during another (e.g. static analysis).
In which case "static analysis" would no longer be static.
I had thrown together something very similar using s7 for a super small self-contained preprocessor.
I'm using `@` as the symbol to consume the next-sexp and process it, otherwise it's mostly the same.
I'll try to throw together some documentation, examples and a LICENSE in case anyone wants to use it. But I've just been using it internally as a generator for Ninja files for my builds.
We have this at work, actually. I wrote a macro language for pascal in guile. At first it was a first draft, but it worked well enough and got to stay.
It generates enormous amounts of code that would otherwise have been unmaintainable. As it is now, it is just a couple of hundreds of lines of scheme and pascal code that recursively generates pascal.
We had some problems making it scale (as I said, it generates a lot of code), but the upcoming 2.2 guile solved most of our issues, and I rewrote parts of it to make it fast enough. Got it down from 1 minute to 10 seconds on a 10+ minute build when rebuilding everything.
I might be able to opensource it. With minor tweaks, it can probably be made a bit more language-agnostic.
32 comments
[ 3.1 ms ] story [ 51.7 ms ] thread> How easy was that!!?
;)
The problem comes mostly from lisp's dislike for text (my personal opinion). Lisps work best when everything is a list, always. As soon as you cross the boundary into... Well.. Anything else , really, you need a little bit of ugly glue.
That's why it's so ugly, in case anyone was about to make the "ermagerd if lisp is so great explain this..." comment. :)
[edit: which, btw, is not to lisp's credit, imho.]
Also, the example could and probably should be rewritten more cleanly. Hence the comment. That was the first thing I used the tool for and my scheme has improved since.
In other words, what is a concrete example of list syntax being inadequate? Literally every data format of which I can think can be expressed as a list or a derivative of lists (e.g. association lists).
I never write those import statements by hand anyway, that's one of the many reasons I have an IDE.
I don't know what IntelliJ has inside of it, but I know Eclipse is based on EJC, the Eclipse Java Compiler, which can turn Java code into a Java abstract source tree, and then turn that either back to Java code or to byte code.
It's not too different from the read function in LISP but it is a bit more complex...
That being said I use code generation instead of the IDE all the time but for harder things like generating fluent builders , copying objects, generating visitor pattern for pattern matching etc.
What I do is write Groovy scripts and will either use Groovy GString templates or Squares Poet to generate code.
What I do though instead of making some sort of DSL is use a subset of the language itself to help the code generator. For example copying a list of private fields I can quickly turn into either an immutable object with builder or an interface or just a plain object copy etc etc.
I put my Groovy scripts in my ~/bin and then just keep my terminal running. I then copy from Eclipse/Intellij some code and then run:
Then paste back the code into the IDE. Ideally I would use APT libraries (and in same cases I have converted Groovy code into APT libraries) but some times I just want something lightweight.Agreed that the example is weak. I do agree that, for import statements, having and IDE fill those in for you is fair game. It can figure out what you are trying to use and offer to import those, that's fine.
The problem here is the slippery slope. You get an IDE writing import statements. Than it will write getters and setters for you – which is already an aberration that prevents language evolution. For instance, C# has a syntax for default getters and setters (and for properties, in general).
Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet).
It's amazing the amount of machinery that gets created when people do not have access to a useful macro system.
It would have been interesting if you had for example generated byte code or enhanced some existing code (Java classes) but I realize that was is probably out of scope for the article. For example it would be funny/interesting if you had Guile VM generating JVM byte code in realtime and some how the Guile VM could figure out some sort of optimizations based on macros to generate better bytecode.
A lot of people don't realize that Java offers a macro/preprocess like system (APT) and it is even sort of hygienic (an example project is Google's Auto [1]).
APT is not that powerful but it gets you quite far and going back to OCaml the new ppx language extension is actually fairly similar (ie annotation processing instead of a complete macro/preprocessing language). .NET has something analogous but even more powerful (IIRC as well as the whole access to the AST aka LINQ style).
> Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet).
2001 called and wants it example back :) . Yes APT didn't exist for a while. Tangental but XDoclet sort of reminds me of Go's very weak annotation system. And again It takes time to get a macro/preprocessing engine right particularly for typed languages. e.g. Rust has had some fair challenges getting this right and even their macro system is not completely hygenic IIRC.
[1]: https://github.com/google/auto
I don't think that types are the issue but rather that such languages were not designed with macro systems in mind from the very beginning. Some added complexity is unavoidable, of course.
> A lot of people don't realize that Java offers a macro/preprocess like system (APT) and it is even sort of hygienic (an example project is Google's Auto [1]).
I didn't. Will look it up.
> 2001 called and wants it example back
I had to use that abomination as far as 2005. I understand that's a dated example, but it is the best example of that slippery slope taken to its extremes that I could think of. It is also a good example in the sense that it used XML instead of expressing itself in the original language. As such, it is also a nice illustration of Greenspun's Tenth Rule.
[1]: https://docs.oracle.com/javase/7/docs/api/java/lang/instrume...
I thought I had a reference somewhere to back that up but can't find it. However I do massively agree having had better planning early in the language could have prevented some serious headaches (its one of the reason I have concerns about Go).
That being said there are some that think macros are inherently code smell particularly if your language has lazy evaluation. e.g. Haskell programmers can go quite far with out having macros. That is controlling evaluation may free the need of a majority of macros (ignoring the verbosity saving aspects).
- A "real" programming language (Scheme)
- Only 1 character to avoid/escape (~)
- Expression-oriented (as opposed to, for example, impure echo/print)
- Completely orthogonal/independent of the code being manipulated (unlike the myriad Java/JVM examples cited by other comments)
One thing that would be nice is to distinguish between pure expressions and impure ones. That way, pure expressions can always be processed automatically (e.g. in wrapper scripts around tools like static analysers) since they're incapable of performing any effects.
Impure expressions would still be useful, e.g. to scan filenames in the current directory, but can be identified as such and skipped by tools/scripts.
The problem is that impure code can do dangerous things (depending on which i/o primitives are provided). For example, a macro might create and delete a bunch of temporary files; this might work fine in one scenario (e.g. during compilation) but misbehave during another (e.g. static analysis).
In which case "static analysis" would no longer be static.
I had thrown together something very similar using s7 for a super small self-contained preprocessor.
I'm using `@` as the symbol to consume the next-sexp and process it, otherwise it's mostly the same.
I'll try to throw together some documentation, examples and a LICENSE in case anyone wants to use it. But I've just been using it internally as a generator for Ninja files for my builds.
https://github.com/Arelius/s7pp
It generates enormous amounts of code that would otherwise have been unmaintainable. As it is now, it is just a couple of hundreds of lines of scheme and pascal code that recursively generates pascal.
We had some problems making it scale (as I said, it generates a lot of code), but the upcoming 2.2 guile solved most of our issues, and I rewrote parts of it to make it fast enough. Got it down from 1 minute to 10 seconds on a 10+ minute build when rebuilding everything.
I might be able to opensource it. With minor tweaks, it can probably be made a bit more language-agnostic.
Maybe I'll use this to hack generics into Go …