The macros run at compile-time, and emit code by use of the <[ ]> blocks -- those get turned into ASTs at compile-time and get filled with the proper data.
Julia is not lisp like, and has macros [1]. The way it works is that your code is first interpreted to get the AST that looks like LISP code (`a+bc+1` becomes `+(a,(b,c),1)`) and your macro operates on that. So basically it works like a lisp macro (until you want to create a macro that creates macro, which is something I haven't done but heard about).
Haskell's "Template Haskell" system is pretty sophisticated... And challenging enough to be some deep black magic.
You end up being able to reify the entire Haskell syntax tree from "quoted" fragments, manipulate and generate a modified tree, and then "print" it back into program flow.
Totally capable, but not terrifically fun. It also has some compilation restrictions which are annoying if necessary.
I just (last night) dug into some TH hackery. I put together a quasiquoter for testing/debugging that would walk a do block and decorate every statement with (`onException` ...) set up to print the source location when an exception is thrown. I actually had to first parse with haskell-src-exts to get the location info, and currently I'm turning it into an ExpQ by pretty-printing and reparsing - which I don't love, but gets the job done.
TH also ruins Haskell by inserting inivisble code with spooky action at a distance, where you have to pre-import symbols that TH will insert later, and compiler errors refer to code defined somewhere else but print line numbers of where the generated code is inserted. TH needs a compiler!
Sweet.js (http://sweetjs.org) and MacroPy (https://github.com/lihaoyi/macropy) come to mind. Actually "real macros" are just macros which operate on AST instead of on strings, they are hard to do if you have to re-implement language parser, but rather easy if the language exposes its AST api, like in Python case.
The title is misleading. By itself, it seems to imply that the author has something against the practice of metaprogramming in general, rather than just Ruby metaprogramming.
Unless I'm building a framework, should I really care?
I'm a "mort" line of business apps dev (well not just that, I do lots of other things, but hey). Do I really need to care about metaprogramming, my gut instinct says no because our business and customer requirements change all the time, and unpredictably. And if I was spending time thinking about "meta programming", nothing would get done.
I would think of it as more code that generates code, or functions that get called at compile time rather than run time.
I've had code that I copied and pasted, and changed, but couldn't factor out into a separate function easily. In those cases, a macro to write the code would have worked nicely.
No offense, but I think your comment is a bit short-sighted. Meta-programming is just another tool in your tool-belt. It's not going to take you 6 months of practice to "get it".
I don't disagree with you, but some people might actually have problems 'understanding' metaprogramming; the same way some people have difficulties understanding pointers, or working through TAoCP/SICP/HTDP.
Problem is, once you get it; it's unfathomable that you might not understand it.
I imagine if you spent most of your time worrying about language design in general, your "business apps dev" wouldn't be too successful. Some people look a little further than the meal on their own plate.
It's my experience, even in boring business code, that learning more techniques can significantly help. Common OO/procedural code has a ton of boilerplate that can be eliminated with functional and metaprogramming techniques, for instance.
But you're right, you don't need to learn anything. Facebook is doing well and it was written in PHP. I hear many popular books were written on typewriters, or even with pen and paper.
Does anyone have some really good examples of metaprogramming? I bought a book on Ruby Metaprogramming and the examples seemed to be all about disguising a string parameter as a member name. So for instance:
foo.somefield
Versus:
foo["somefield"]
And variations on that theme. While it's cute, I guess (you're not type checking anyways so nothing lost, I suppose), I don't really get it.
I've seen that before, but isn't that the same theme? db.find_by_userid(123) versus user.find_by("userid", 123)?
If everything is checked dynamically, does it really make a difference beyond the 3 characters? Sure find_by_userid looks somewhat nicer, for sure. But is that really it? Any examples that aren't trivially replaced by a string param (or two)?
The CanCan ruby authorization library does a neat bit of metaprogramming. It offers a function "load_resource" which you put at the top of your controller. The function uses the name of the controller class and the name of the currently executing controller action to automatically set a resource. It looks like this:
class CarsController < ActionController::Base
load_resource
def index
# @cars is defined
end
def show
# @car is defined
end
end
I don't know what qualifies as "really good examples" but there are lots of random things that use metaprogramming in some capacity and are popular. Lots of IDLs, serialization formats, ORMS etc. I am thinking stuff like Protocol Buffers, Hibernate, Swagger, etc.
I dunno, it seems like metaprogramming seems to refer to something more powerful than just reflecting over definitions, which is mainly what those things need. Although some ORMs emit runtime code.
I guess my question is better phased as "Ruby goes on a lot about metaprogramming. I've used .NET reflection quite a bit. What does Ruby's metaprogramming really enable that I'm missing out on?"
I actually like Ruby's limited and verbose meta-programming. It's just painful enough that when I get the urge to reach for it, I think just that much harder about ways to go about what I'm trying to do without it.
And the methods themselves, while verbose and clumsy, are actually fairly transparent and intention-revealing. I understood perfectly what the article author was trying to do and why.
Ruby is expressive enough that you usually don't need it, and when you do need it, the stark stylistic difference between the normal methods and the meta-programming code feels safe and comforting. You see `define_method` or `Class.new` and your brain instantly reads it as magic and start wondering what the hell caused the programmer to start delving into the hard stuff.
You almost never see meta-programming in Ruby where it wasn't needed, and it's almost always limited to solving one or two problems.
In my own brief foray into C#, it wasn't long before I found a problem that, when brought to a more experienced coworker, he solved with reflection. I couldn't remember the details, but I knew that had I had to solve it in Ruby, I'd never need anything like that.
Would you be happy with examples in other languages? I think using meta-programming to add hash literals to CL is a rather good example: http://frank.kank.net/essays/hash.html
There's a chapter in pg's "On Lisp" about continuation-passing-style transformation (from sequential code to CPS) done with meta-programming. Worth reading, too.
If you're an Emacs user, the source code of EIEIO is a good read. It's an object system implemented on top of Elisp using it's meta-programming capabilities.
In Smalltalk-land the most impressive example of meta-programming is certainly a debugger. You get to inspect the state of objects while they are running, you can modify it, and you can modify method bodies while they are running. Scary stuff.
In Python some nice meta-programming code lives in Django ORM, in Model metaclass, and in Fields. This is the code which allows you to declaratively define your models as classes. Similar code is used in Django Forms.
The talk "Five years of bad ideas" by Armin Ronacher is a 40-minuted tour of some of the most advanced meta-programming available in Python.
In JavaScript every library which implements "classical classes" on top of objects and prototypes uses meta-programming. You can read about it more in depth in Raganwald "the art of javascript meta-object protocol" and his other talks and posts.
Angular.js dependency injection framework is based on meta-programming; actually I had a mixed reaction when I saw the code for this, where they rewrite a user supplied function as a string and compile a new function out of this string. It's a good example of meta-programming, but it's also very crude compared to hygienic macros in Schemes for example. It lives here: https://github.com/angular/angular.js/blob/master/src/auto/i...
Generally, meta-programming is a vast and varied set of techniques. It's worth knowing about them, because most of the time you don't need them, but when you do, you really, really need them very much.
Yeah, I don't get the unity of it all, or why it's called "metaprogramming". The examples always look like they fall into one of two categories:
1. super-easy development of new parsers/compilers (lisp macros)
2. shenanigans with dynamicism and first class functions.
Both of those are cool! But I can't figure out what they have to do with each other, other than that you don't get much of that sort of thing in Java/C.
Think about it this way: what can you do to code, with code? You can create new code (simple macros, emitting optimized bytecode, etc.) or you can modify existing code (self-modifying code, instrumentation (example: Quasar and Pulsar in Java http://blog.paralleluniverse.co/2013/05/02/quasar-pulsar/), etc.) and that's about it. Exactly how you go on about this varies wildly depending on what tools you have at your disposal, but the unifying thing about all of them is that you're writing code which acts on other code, for some (rather fuzzy) definitions of "code" and "acting on".
One nice example of modifying code on the fly I just remembered is goto decorator in Python: http://code.activestate.com/recipes/576944-the-goto-decorato... As for code generation, if I recall correctly, Salt (http://docs.saltstack.com/en/latest/) uses YAML and Jinja2 to generate shell scripts. These are very different techniques, but the thing they have in common is that they manipulate code, which makes them both examples of meta-programming.
Thanks a lot! I suppose I was getting caught up in a limited definition of metaprogramming, as essentially every Ruby example I've gotten is "we moved a parameter to the method name".
The LISP macros make me jealous. What non-LISP-like languages have macros? Nemerle does and looks fairly neat. It'd seem like somehow passing the AST to user-defined modules would get ya pretty far, but there must be a ton of complications, otherwise more languages would offer it.
I suppose we can consider converting quoted code to another target (like GPU, or database) as metaprogramming. I guess I had filed that under "just reflection".
That could be the subject of many volumes. Most popular hypotheses:
- corporate sponsorship dried up
- it wasn't open source, and by the time squeak came around, usage had dried up considerably
- it was actually too hard to learn
Although, it's unfair to judge it by today's standards. The sheer number of open source programmers just didn't exist when Smalltalk was popular.
You see some vestigial remnants of smalltalk in modern languages (in ruby, everything is an object, dart allows you to snapshot the object memory, the idea of a virtual machine, etc, etc) but having all of them together made smalltalk quite groundbreaking. We haven't really learned all the lessons quite yet, either. (Just my opinion). One of the biggest misunderstandings was that object oriented programming is really about message passing, not about encapsulation.
I'd add the "closed world"/"walled garden" nature of image-based Smalltalks as one more popular hypothesis. On the other hand I never heard the "too hard to learn" argument: highly regular and small syntax coupled with very interactive experience of ST browsers and other tools made it a joy for me to learn. I have only one complaint: as I learned Pharo I frequently tried to learn some interesting package but I couldn't find the "entry point" to it. It took me a long while to realize how to use Pharo's FileSystem library, just because it wasn't obvious what the external interface is supposed to be.
Anyway, Smalltalk still exists and not only as an inspiration for Ruby. Pharo and Squeak are very nice image-based environments, VisualWorks has "enterprise grade" support, GNU Smalltalk is small and lightweight and works well for scripts. There is Amber, which tries to recreate Smalltalk image inside the browser and there's (forgot the name) which tries to implement ST on JVM (it was on kickstarter I think). There's also Newspeak, which is highly experimental, and LivelyKernel (by Dan Ingals) which again transplants some of the more important ST ideas into the browser. And all that on top of mainstream languages gaining more and more features which Smalltalkers are used to. It's certainly an interesting time to be a part of Smalltalk community.
I'd say the whole game of eDSL creation is an act of metaprogramming. There are examples littered everywhere in the Haskell world—though rarely do syntax abstractions like macros get used. This is because it's relatively easy to embed a natural style program in a monad or applicative functor, access the syntax, and compile it.
While the link focuses on Ruby, I think there's a more general reason (which also explains why more languages don't support a lisp-like macro): cooperative development.
When you're writing code in a vacuum, you can fill it with all the metaprogramming you want. You probably have some preferred style, and your metaprogramming will reflect that. In something like lisp, this gets taken to the extreme -- code can look very different from what traditional lisp looks like when there are macros involved.
While this is great for developing solo, or with a small group, it becomes too much to handle as more people onboard. Every new contributor needs to learn your particular style, and how your macros work, and then how to apply them effectively. Its much easier when new functionality and expressiveness is added through a common format (like adding a new method).
As a lisp fan, those methods feel clumsy to me. But if you want to do something that's so big you need more contributors, it's worth the tradeoff.
Does it really matter in a dynamic scripting language? When using reflection in a strong programming language there is a chance it will fail at runtime instead of being checked at compile time. Ruby only fails at runtime, so as long as the code is shorter it should not be a much bigger problem than it already is.
37 comments
[ 3.7 ms ] story [ 97.4 ms ] threadThe macros run at compile-time, and emit code by use of the <[ ]> blocks -- those get turned into ASTs at compile-time and get filled with the proper data.
https://news.ycombinator.com/item?id=7071429
http://docs.julialang.org/en/release-0.2/manual/metaprogramm...
You end up being able to reify the entire Haskell syntax tree from "quoted" fragments, manipulate and generate a modified tree, and then "print" it back into program flow.
Totally capable, but not terrifically fun. It also has some compilation restrictions which are annoying if necessary.
https://github.com/dlthomas/snowdrift/compare/5094b1ca0e5002...
http://metalua.luaforge.net/quicktour.html
http://infraruby.com/blog/ruby-metaprogramming-with-s-expres...
http://infraruby.com/blog/the-structure-of-s-expressions
http://infraruby.com/blog/generating-an-html-generator
I'm a "mort" line of business apps dev (well not just that, I do lots of other things, but hey). Do I really need to care about metaprogramming, my gut instinct says no because our business and customer requirements change all the time, and unpredictably. And if I was spending time thinking about "meta programming", nothing would get done.
Problem is, once you get it; it's unfathomable that you might not understand it.
But you're right, you don't need to learn anything. Facebook is doing well and it was written in PHP. I hear many popular books were written on typewriters, or even with pen and paper.
If everything is checked dynamically, does it really make a difference beyond the 3 characters? Sure find_by_userid looks somewhat nicer, for sure. But is that really it? Any examples that aren't trivially replaced by a string param (or two)?
I guess my question is better phased as "Ruby goes on a lot about metaprogramming. I've used .NET reflection quite a bit. What does Ruby's metaprogramming really enable that I'm missing out on?"
And the methods themselves, while verbose and clumsy, are actually fairly transparent and intention-revealing. I understood perfectly what the article author was trying to do and why.
Ruby is expressive enough that you usually don't need it, and when you do need it, the stark stylistic difference between the normal methods and the meta-programming code feels safe and comforting. You see `define_method` or `Class.new` and your brain instantly reads it as magic and start wondering what the hell caused the programmer to start delving into the hard stuff.
You almost never see meta-programming in Ruby where it wasn't needed, and it's almost always limited to solving one or two problems.
In my own brief foray into C#, it wasn't long before I found a problem that, when brought to a more experienced coworker, he solved with reflection. I couldn't remember the details, but I knew that had I had to solve it in Ruby, I'd never need anything like that.
There's a chapter in pg's "On Lisp" about continuation-passing-style transformation (from sequential code to CPS) done with meta-programming. Worth reading, too.
If you're an Emacs user, the source code of EIEIO is a good read. It's an object system implemented on top of Elisp using it's meta-programming capabilities.
In Smalltalk-land the most impressive example of meta-programming is certainly a debugger. You get to inspect the state of objects while they are running, you can modify it, and you can modify method bodies while they are running. Scary stuff.
In Python some nice meta-programming code lives in Django ORM, in Model metaclass, and in Fields. This is the code which allows you to declaratively define your models as classes. Similar code is used in Django Forms.
The talk "Five years of bad ideas" by Armin Ronacher is a 40-minuted tour of some of the most advanced meta-programming available in Python.
In JavaScript every library which implements "classical classes" on top of objects and prototypes uses meta-programming. You can read about it more in depth in Raganwald "the art of javascript meta-object protocol" and his other talks and posts.
Angular.js dependency injection framework is based on meta-programming; actually I had a mixed reaction when I saw the code for this, where they rewrite a user supplied function as a string and compile a new function out of this string. It's a good example of meta-programming, but it's also very crude compared to hygienic macros in Schemes for example. It lives here: https://github.com/angular/angular.js/blob/master/src/auto/i...
Generally, meta-programming is a vast and varied set of techniques. It's worth knowing about them, because most of the time you don't need them, but when you do, you really, really need them very much.
1. super-easy development of new parsers/compilers (lisp macros)
2. shenanigans with dynamicism and first class functions.
Both of those are cool! But I can't figure out what they have to do with each other, other than that you don't get much of that sort of thing in Java/C.
One nice example of modifying code on the fly I just remembered is goto decorator in Python: http://code.activestate.com/recipes/576944-the-goto-decorato... As for code generation, if I recall correctly, Salt (http://docs.saltstack.com/en/latest/) uses YAML and Jinja2 to generate shell scripts. These are very different techniques, but the thing they have in common is that they manipulate code, which makes them both examples of meta-programming.
At least that's how I understand the term :)
The LISP macros make me jealous. What non-LISP-like languages have macros? Nemerle does and looks fairly neat. It'd seem like somehow passing the AST to user-defined modules would get ya pretty far, but there must be a ton of complications, otherwise more languages would offer it.
I suppose we can consider converting quoted code to another target (like GPU, or database) as metaprogramming. I guess I had filed that under "just reflection".
- corporate sponsorship dried up
- it wasn't open source, and by the time squeak came around, usage had dried up considerably
- it was actually too hard to learn
Although, it's unfair to judge it by today's standards. The sheer number of open source programmers just didn't exist when Smalltalk was popular.
You see some vestigial remnants of smalltalk in modern languages (in ruby, everything is an object, dart allows you to snapshot the object memory, the idea of a virtual machine, etc, etc) but having all of them together made smalltalk quite groundbreaking. We haven't really learned all the lessons quite yet, either. (Just my opinion). One of the biggest misunderstandings was that object oriented programming is really about message passing, not about encapsulation.
For more: http://gagne.homedns.org/~tgagne/contrib/EarlyHistoryST.html
Anyway, Smalltalk still exists and not only as an inspiration for Ruby. Pharo and Squeak are very nice image-based environments, VisualWorks has "enterprise grade" support, GNU Smalltalk is small and lightweight and works well for scripts. There is Amber, which tries to recreate Smalltalk image inside the browser and there's (forgot the name) which tries to implement ST on JVM (it was on kickstarter I think). There's also Newspeak, which is highly experimental, and LivelyKernel (by Dan Ingals) which again transplants some of the more important ST ideas into the browser. And all that on top of mainstream languages gaining more and more features which Smalltalkers are used to. It's certainly an interesting time to be a part of Smalltalk community.
A smattering:
http://hackage.haskell.org/package/accelerate https://github.com/mainland/nikola http://hackage.haskell.org/package/esqueleto http://hackage.haskell.org/package/api-tools http://hackage.haskell.org/package/atom http://hackage.haskell.org/package/diagrams http://hackage.haskell.org/package/c-dsl https://hackage.haskell.org/package/js-good-parts http://hackage.haskell.org/package/Bang http://hackage.haskell.org/package/orc
And if you buy that definition, then there is some ungodly metaprogramming available in dependently typed languages.
https://github.com/pigworker/MetaprogAgda/blob/master/notes....
Though, to be clear, there are few who can follows those notes today.
When you're writing code in a vacuum, you can fill it with all the metaprogramming you want. You probably have some preferred style, and your metaprogramming will reflect that. In something like lisp, this gets taken to the extreme -- code can look very different from what traditional lisp looks like when there are macros involved.
While this is great for developing solo, or with a small group, it becomes too much to handle as more people onboard. Every new contributor needs to learn your particular style, and how your macros work, and then how to apply them effectively. Its much easier when new functionality and expressiveness is added through a common format (like adding a new method).
As a lisp fan, those methods feel clumsy to me. But if you want to do something that's so big you need more contributors, it's worth the tradeoff.