Well the example _is_ building a brainfuck interpreter. I guess this is the perception that comes with having minimal syntax like this, but it's probably also a functional requirement. Would recommend people read the whole thing before putting it off as a joke.
Indeed. I am the writer of this blog post and Matthew and I never intended
for this to be a joke. In fact, we thought we had programmed something unique
that nobody had done before, and the bootstrapping code syntax is minimal by
design (you don't want to assume a default complicated syntax).
Since you're here, note that I can't resize your text on Chrome on Android by default, and so I can't read your page on my phone at all without my glasses without switching the browser to desktop mode.
i think you might lose people in the transition from the turing tape machine style bootstrapping to using full "words" for operations. it might help to compare it to something like regexes? (obviously they're not similar in the way that forth / stack langs are, but they're more commonly known and more commonly thought of as useful)
i think it'd also help a lot to have some gifs of the state machine operating on stuff
or maybe you could have something with "syntax" highlighting. a navigable timeline that changes the highlighting based on the current ignore lists and / or stack contents as you step through the code could be neat
As mentioned elsewhere I think they do themselves a disservice by starting there, instead of starting with a reasonable set of delimiters etc., and then referring to an aside that explains how to bootstrap to that point. The lowest levels get too esoteric for most.
It's a bit like being taught functional programming by starting with composing everything from combinators, including church numerals for numbers. Sure, knowing those bits is worthwhile theory, but not until you're convinced about the higher level utility.
Laughed out loud when I scrolled far enough to see an actual example. Still not sure if this is a subtle satire about people’s aversion to Lisp syntax. The ideas here are really interesting though, of pushing metaprogramming further up the chain to tokenisation not just parsing. Either way, I hope it’s a compliment to say this could easily have appeared on Aphyr’s blog.
The entire point of this exercise appears to be this: Forth has one character you cannot redefine, the space, what happens if we remove that restriction?
The part of the bootstrap you're referring to is, in fact, the part where they tell the reader to treat space and newline as delimiters, so it appears you're complaining about whitespace being significant in the part of the program where they declare it to be a delimiter.
Such is your right, of course, but it does lead one to wonder if you had some better idea for how to go about this.
Hi, I'm the one that wrote this blog post! A lot of comments treat this project
as a joke or some form of satire, but we've spent 3 months writing this after
initially coming up with the idea because we thought nobody had created such
a general programming language before. It's good to get some feedback and
maybe we've been in our own heads for a little too long, but it's good that
someone gets it.
I read it as probably containing something deep that I should save for when I have time to go through the details. Maybe we syntaxlings need time to get comfortable.
> we thought nobody had created such a general programming language before
How is this more general than a language like Binary Lambda Calculus, which can bootstrap into any other language, like Brainfuck in 112 bytes of bootstrap, or even into Cognition itself?
Writer of this article: Thank you! I don't mind the snakry comments. In fact, I welcome them as I think they are pretty funny themselves. We'll most certainly be working on more dark magic in the future.
you're supposed to touch a few nerves. that means you're showing something different to the comfort of most. something really different. two most common reactions to this is irritation and laughter.
This is really cool! For those who take it as a joke or who scoff at the choices (like the whitespace thing), at least read until section 4 and then skip to section 9.
Regarding the "whitespace thing" I think they're doing themselves a disservice by not starting with predefined rules allowing them to use whitespace delimited words and deferring showing how you can bootstrap to that point until later.
In general, show more of what is possible in a more accessible way first, before presenting "line noise".
Overall that document would be helped by some editing to bring out more of what a reader who haven't spent months thinking about this will find interesting first.
ding! I think the bootstrapping code might've bitten us. Of course there's no
practical reason why we _needed_ to not have the delim flag flipped and a couple
of delimiters defined, but there's a conceptual symmetry in it being the way
it is. Still, I agree in large part with your point.
It seems as though you've read the entire article and understood a decent
portion of it. I'm impressed because I think I explained this suboptimally.
I think it's fine to show that you can do it, I think the main thing is to flip the order a bit.
E.g. "here's a cool thing thing we can do <demonstrate the outcome of significantly changing a readable syntax>" to hook people, "here's how <show how you change syntax with higher level helpers>", "and if you really want to know how to bootstrap this from basics <here comes the linenoise>".
Maybe compare how e.g. Forth is often introduced, with how people describe bootstrapping of a simplistic Forth like Jonesforth or Sectorforth [2]. Showing people how they can define their own words and it fundamentally changes how they work with the language afterwards is cool to a lot of people who have no interest in details like how you an implement even numbers with a minimal set of primitives (e.g. Sectorforth relies on that - it doesn't have builtin numbers[3]).
Both are interesting to me, but I'm weird, and I think for most people it'd be easier to maintain their interest if those two aspects are either separate articles or at least if the bootstrapping is relegated to a standalone section they're clearly told they can skip.
[3] The Sectorforth Hello world defines every numeric constant it needs like this:
: -1 ( x -- x -1 ) dup dup nand dup dup nand and ;
: 0 -1 dup and ;
: 1 -1 dup + dup and ;
: 2 1 1 + ;
: 4 2 2 + ;
: 6 2 4 + ;
Which is fun if you're a language geek. Not so convincing if you want to know if Forth is fort you [EDIT: That mistake was wholly unintentional, but I'll leave it].
>This is of course true, but there's something very broken with lisp: metaprogramming and programming aren't the same thing,
Metaprogramming and programming are the same thing. It's just that no language, including all lisp, (but hilariously not m4) get quotation wrong. Lisp gets around this with macros which let you ignore quotation and deal with meta language statements expressed as object language statements when they clearly should not be.
This issue stems from the fact space in the object and meta language is treated as the termination of an atom without distinction between the two.
>Cognition is different in that it uses an antisyntax that is fully postfix. This has similarities with concatenative programming languages
Postfix languages are a dual of prefix languages and suffer from the same issue. You either need to define the arity of all symbols ahead of time and not use higher order functions or you need a pair of delimiters which can serialise a tree. Relying on an implicit zeroth order stack solved the problem in the same way a lobotomy solves depression.
Thanks for your feedback! I'm not accusing you of not reading the full article, but you should if you haven't already. Also, we don't know the extent to which we made anything new; if you think you can do what we're doing in lisp in some way, you're free to prove us wrong.
> Relying on an implicit zeroth order stack solved the problem in the same way a lobotomy solves depression.
I mean, fair and flowery, but an implicit stack has been a thing ever since proto computers/calculators. Just like a lobotomy reduces the availability of higher order processing, so too does going back to the utmost primitives in calculating a string of instructions
Wow. The proverb of "Languages that don't change your way of thinking about programming are not worth learning" in full swing here, I feel like I've been enlightened.
This is a practical example of bootstrapping a minimal machine into an interpreter for a high level language. As I remember it being taught to me the importance of being able to do this — with a Turing machine, lambda calculus, etc. — is to show that the high level language is equivalent to the fundamental language and therefore whatever can be reasoned about the former can be applied to the latter.
The first and only example I can think of is the halting problem. On a practical scale one might prove that the base language doesn’t have memory leaks, so neither can the derived language? What are the advantages of bootstrapping like this? If the answer is simply another form of “because it’s there” (Mallory, re climbing Everest) then I respect that!
I'm not persuaded this is a better idea than, say, Racket's ability to configure the reader layer [1]. This lets you create, for example, an embedded Datalog implementation [2] that uses Datalog syntax but interops with other Racket modules. (The underlying data model doesn't change.) This gives you the ability to metaprogram without being confined to S-expressions, but it does so in a high-level way.
It is very neat to see this kind of syntax bootstrapping. I think there's some value (in a researchy-sense) to being able to do that. But I'm not sure if there's something fundamentally "better" about this approach over Racket's approach.
Postscript: Lisp (and Scheme and Racket) macros typically operate on AST (typically because Lisp has reader macros and Racket has a full-bodied reader extension) but Rhombus [3] operates on a "shrubbery", which is like an AST but it defers some parsing decisions until later. This gives macros a little flexibility in extending the syntax of the language. Another interesting point in the design space!
[3]: Flatt, Allred & Angle et al. (2023-10-16) Rhombus: A New Spin on Macros without All the Parentheses, Proceedings of the ACM on Programming Languages. https://doi.org/10.1145/3580417
As a side effect, we can see that the macro gets a simple list passed. A list of numbers and a symbol ("symbol" is also a data type). Not text. Not an AST.
Also data which is possibly not read, but computed by other code. We can compute the arguments of the macro REV and pass the thing to EVAL. Works the same as above.
CL-USER 10 > (eval (append '(rev) (loop for i from 1 to 4 collect i) '(+)))
(1 2 3 4 +) is a LIST
0 1
1 2
2 3
3 4
4 +
10
The macro only gets that data and can do anything with it. There is no idea of Abstract Syntax Tree: it does not need to be a tree, it does not need to be valid Lisp code and it carries no syntactical information. Generally, it also does not need to return valid Lisp code. To be computed all we need is that the evaluator eventually sees valid Lisp code, not the individual macro.
In Lisp the "reader" by default only parses a data layer: symbolic expressions. EVAL, macros and other Lisp functionality gets mostly data passed. EVAL has to figure out, what (a b c) actually is as a program. It can traverse it in an interpreter or compile it. A compiler may internally create AST representations -> but that free to the implementation.
The Lisp language then typically is not defined over text syntax, but data syntax.
A Lisp interpreter processes during execution not text, but s-expressions. It's a "List Processor", not a Text Processor. Lisp not Texp. ;-) The function COMPILE gets an s-expression passed, not text.
It usually is a tree, just not a syntax tree. For example, the "if" special form in CL has the syntax:
(IF <text-expr> <then-expr> [<else-expr>])
But appearing in a macro expansion, it would just be a list where the first item was the symbol "IF" from the "COMMON-LISP" package. In addition, you could put e.g.
(IF foo bar baz biff quux)
Inside the body of a macro you would get a list that does not match the syntax of the "if" special form, despite superficially looking like such a list. This doesn't match anything that one would call an "AST" in other languages, which would enforce syntactical correctness of special forms.
From my limited understanding, it seems like lispers means AST in an ad-hoc way. There's no statically predefined solid structure describing what an if-node or a lambda-node is. We all agree to squint and look at (<idea> <parameters-potentially-recursive>) as an ast in spirit.
Scheme did implement actual syntax objects and other lisps may have a concrete ast (pun slightly intended) layer but it's not required to enjoy the benefits of sexp as code and data.
An "AST in spirit" is in our mind, but not the machine.
(let ((s (s s)))
(flet ((s (s) (declare (type s s)) s))
(tagbody (s) s (go s))))
The function READ has no idea what S is: variable, operator name, data symbol, type, go tag, function name, macro name, ...? For each above we don't know what s is. We would need to parse it according to some syntax (and possibly know the current state of the runtime.
An AST would be the product of a parser and the parse would parse the code above according to some provided syntax. The AST then would encode a tree built from some syntax, where the nodes would be classified. In Lisp symbols and lists have multiple uses and the s-expression does not encode which uses it is: is it data, is it some kind of syntactical element. There is also no defined parser and no syntax it encodes on that level. READ is at best an s-expression reader, where it has zero knowledge what the lists and symbols supposed to mean in Lisp. For the reader (+ a b), (a + b), (a b +) are just s-expressions, not Lisp code.
I mostly agree, but it seems (I can only speak as a spectator/reader) that this lack of information was not a deal breaker for the community or even producing large systems. The type resolution will happen at use-site, and if `s` is not existent where it should be (a binding in the environment, a tag in the right namespace) or not what it should be, your code fails and people will adjust.
Where there serious defects caused due to this dynamic nature (honest question) ? it seems to me that people adjusted to this without big troubles. Not that I'm against any improvement.
The Lisp compiler will need to figure it out and complain about various problems: undefined function/variable/tag/..., argument list mismatch, type mismatch, etc.
The compiler will expand the macros at compile time and the generated source code then is checked.
One thing this macro system enables are macros which can implement relatively arbitrary syntactical extensions, not restricted by a particular syntax. That can be seen as useful&flexibility or as potential problem (-> needs knowledge and discipline while implementing macros, with the goal of ensuring the maintainability of the code).
You are correct and the precision is appreciated. However your emphasis is somewhat misleading.
In the lisp world, when data represents code, it's probably stored in a tree. Elsewhere, when data represents code, it's probably stored as an array of bytes. There code may not be recognised as being data at all.
It's the difference between eval taking a byte string and taking a parse tree. Rather literally.
This is only a convention. Lisp does it right, but as almost everything else thinks code is strings, confusion abounds.
There is a "data AST" in Lisp. The data AST is source code to Lisp evaluation/compilation. Compilation may produce another AST, but typically this is something internal, and implementation-specific.
The data AST is rich enough to for ergonomic, precise source-to-source manipulation. Even a pretty advanced Lisp compiler can be built which goes straight from the data AST to an intermediate representation, skipping the code AST stage.
"Data AST" means that when we have (+ 1 2), this doesn't say "I'm an arithmetic expression", but rather something weaker: "I'm a list of three elements: a symbol object, and two integer objects".
The list object is an abstract syntax tree for the printed list. It must be. It not a parse tree because a parse tree would preserve the representation of the parentheses: in a parse tree, every grammar symbol appears: the nodes of the tree are 1:1 to the grammar rules that they match. Since the parentheses, and whatnot, are gone, it must be abstract syntax.
Most of Lisp syntax is designed such that its syntactic units correspond to nodes of the data AST. That makes source-to-source transformations ergonomic, because the data AST data structure is easy to manipulate.
I'm not sure that something which uses brainfuck as its default example is intending you to take it seriously. Personally I burst out laughing at the introduction of "metacrank".
Thank you for making this! I've fallen into investigating the limits of language flexibility before, (and got into f-exprs and Shutt's thesis about Kernel, a great time!), but I've never seen anything quite like this.
It's inspired me to mess around in this area again, and I'll definitely have to re-read your blog post a few times.
Unseemly lang is another project that goes the other direction, but I don’t remember if it has balanced brace requirements
https://github.com/paulstansifer/unseemly
Ruby has that... But you're then just implementing your own transpiler or interpreter in full, instead of "just" redefining parts of the grammar. I'm not sold in this approach, but it's an interesting read at least.
I know of multiple ways to do it in perl - 'use Foo;' can introduce a source filter that slurps the entire remaining file text, thus hiding it from the perl compiler.
__DATA__ sooort of counts and is probably the closest to pure STOPPARSE as GP is thinking of.
I also, once, wrote https://p3rl.org/Devel::Declare which basically grabbed the current read position and Did Things, including calling back into the existing perl parser to not have to re-implement everything - that was how perlers first got access to a mostly* working 'method' keyword.
(now there's a proper way of doing that in perl core, and Devel::Declare is (very happily to me) obsolete, but it was essential to prove the concept and the user interest to justify adding the actual feature to the core interpreter)
[*] the caveats are many and believe me I'm well aware the entire thing was a giant hack from the moment I first started writing it, but rather than trying to write them out I'll just say "however hacky you imagine this was, what I actually did is probably worse" and leave it there.
Not really, not as most people (I think) would understand the term.
That just gives you arbitrary input. You can write your own interpreter for a language and do the input in that language, but that’s not “arbitrary metaprogramming” in the host language, at least as most people understand it. Or you can leverage whatever metaprogramming facilities are offered by the host via an external DSL you implement, which may be a more ergonomic way of metaprogramming the host language, but still is limited to the metaprogramming facilities exposed by the host language, so it’s only “arbitrary metaprogramming” to the extent that the host language already exposes arbitrary metaprogramming.
> This is something impossible in other languages, being able to program your own tokenizer for some foreign language from within cognition, and have future code be tokenized exactly like how you want it to be. [Emphases in original]
Doesn't that sound like TeX \catcode? In fact, I think you can bootstrap Cognition from TeX as well.
A little tricky to read this. With the ground constantly changing underneath your feet, there's a feeling that rules and words get introduced then redefined willy nilly. The whole thing has a sense of "Numberwang" about it, which I think is part of why it comes across as satire. Another big part is no doubt how ridiculously the bootstrapping stage was written, but that seems intentional.
There's clearly something deep going on, but I will have to come back to this after an even deeper cup of coffee.
There's a lot to explain, and it has occurred to me that I have explained it in a suboptimal manner. Writer of this post, by the way. The problem is that there is just so much to communicate (the design for this language with Matthew took 3 weeks of back and forth communication over several hours a day). A lot to fill in for people that don't know me in real life.
I’m a non programmer, and while I can often just barely grasp what might possibly be going on in a “normal” language, I can’t even begin to pretend to make sense of what you’ve made. I’m glad some are able to appreciate what you’ve done.
I suspect the reception of it being suspected to be a joke, is the mention of lisp and brainfuck priming them for a joke, combined with examples and concepts that seem to require a much stronger than normal technical background. So for the average Joe it ends up in the “turbo encabulator” zone where it’s not quite clear if what’s going on is real or in jest. The prerequisites for understanding just aren’t there.
I also suspect a non-zero percentage of the readers have involuntarily audio flashbacks of Soulja Boy when they see that many “cranks” on a page.
I thought it was cool if a little impenetrable without extra effort! Reminded me a bit of Urbit’s old theorycraft, which I think is still an open question as to whether it’s an elaborate ruse, so the idea did cross my mind that I’m getting my leg pulled, but it seems to be in earnest. I have some similar thoughts in a related but different domain but user-extensible syntax is certainly a nice and uncommon language feature.
oh yeah, Urbit; I'm quite aware of the project and can recognize that to some I might just be explaining it in an intentionally confusing manner, perhaps. I do think the original intent of Urbit was to naturally select nerds, basically (I mean Curtis Yarvin talking about the old UseNet made it really seem like it), and no, that was not the intent here. It literally is just hard to figure out how to communicate the core concepts of Cognition and why we wanted it to be that way.
I am researching formal methods systems to conduct cryptographic proofs; I think a big issue in that space is having syntax sugar in the core compiler, so I am pursuing ways to move syntactic sugar into a separate macro language.
Your post sounds like an interesting research project, but the examples given in the blog post seem somewhat discouraging to me. When I gaze at one of your examples, they seem obtuse and while I am sure I could understand the examples by not just skimming the article, this does tell me that the grammars you used are not very self-explanatory: Your syntax expresses meaning relative to the specific grammar as opposed to expressing meaning relative to the English language or pre-existing, well-established programming languages.
Is there some example of self-explanatory grammars and self-explanatory bootstrapping code in your language?
Self explanatory grammars: there's the stem-like syntax that I showcase and you could write a lisp like syntax in it pretty easily. I'd imagine you don't want to use this in production yet. As for self explanatory bootstrap code: the bootstrap code is the way it is because cognition doesn't want to assume some complex syntax from the start.
140 comments
[ 3.0 ms ] story [ 187 ms ] threadI think a lot of people on HN nowadays can only see the world through the lens of Javascript and as a result really can't appreciate what this is.
i think it'd also help a lot to have some gifs of the state machine operating on stuff
or maybe you could have something with "syntax" highlighting. a navigable timeline that changes the highlighting based on the current ignore lists and / or stack contents as you step through the code could be neat
It's a bit like being taught functional programming by starting with composing everything from combinators, including church numerals for numbers. Sure, knowing those bits is worthwhile theory, but not until you're convinced about the higher level utility.
thx but no thx
the previous line ends with three space characters to indicate sarcasm; interpret literally wherever trailing whitespace is not readily discernible
The part of the bootstrap you're referring to is, in fact, the part where they tell the reader to treat space and newline as delimiters, so it appears you're complaining about whitespace being significant in the part of the program where they declare it to be a delimiter.
Such is your right, of course, but it does lead one to wonder if you had some better idea for how to go about this.
I don't see how you could do that -without- a literal space being relevant in that way once.
How is this more general than a language like Binary Lambda Calculus, which can bootstrap into any other language, like Brainfuck in 112 bytes of bootstrap, or even into Cognition itself?
That said, personally I think that Forth is as philosophically pure as I'm willing to gaze up the ladder of programming purity. :P
https://aphyr.com/posts/353-rewriting-the-technical-intervie...
Then again I was once a pure math grad, so "beautiful, fascinating, albeit completely practically useless" is something I mean as a compliment.
I need to read up on Stem and then come back and read this post of yours again to better appreciate it, I suspect. But that sounds like fun.
not much is novel nowadays. great job!
In general, show more of what is possible in a more accessible way first, before presenting "line noise".
Overall that document would be helped by some editing to bring out more of what a reader who haven't spent months thinking about this will find interesting first.
It seems as though you've read the entire article and understood a decent portion of it. I'm impressed because I think I explained this suboptimally.
E.g. "here's a cool thing thing we can do <demonstrate the outcome of significantly changing a readable syntax>" to hook people, "here's how <show how you change syntax with higher level helpers>", "and if you really want to know how to bootstrap this from basics <here comes the linenoise>".
Maybe compare how e.g. Forth is often introduced, with how people describe bootstrapping of a simplistic Forth like Jonesforth or Sectorforth [2]. Showing people how they can define their own words and it fundamentally changes how they work with the language afterwards is cool to a lot of people who have no interest in details like how you an implement even numbers with a minimal set of primitives (e.g. Sectorforth relies on that - it doesn't have builtin numbers[3]).
Both are interesting to me, but I'm weird, and I think for most people it'd be easier to maintain their interest if those two aspects are either separate articles or at least if the bootstrapping is relegated to a standalone section they're clearly told they can skip.
[1] https://news.ycombinator.com/item?id=31368212
[2] https://github.com/cesarblum/sectorforth
[3] The Sectorforth Hello world defines every numeric constant it needs like this:
Which is fun if you're a language geek. Not so convincing if you want to know if Forth is fort you [EDIT: That mistake was wholly unintentional, but I'll leave it].Metaprogramming and programming are the same thing. It's just that no language, including all lisp, (but hilariously not m4) get quotation wrong. Lisp gets around this with macros which let you ignore quotation and deal with meta language statements expressed as object language statements when they clearly should not be.
This issue stems from the fact space in the object and meta language is treated as the termination of an atom without distinction between the two.
>Cognition is different in that it uses an antisyntax that is fully postfix. This has similarities with concatenative programming languages
Postfix languages are a dual of prefix languages and suffer from the same issue. You either need to define the arity of all symbols ahead of time and not use higher order functions or you need a pair of delimiters which can serialise a tree. Relying on an implicit zeroth order stack solved the problem in the same way a lobotomy solves depression.
I mean, fair and flowery, but an implicit stack has been a thing ever since proto computers/calculators. Just like a lobotomy reduces the availability of higher order processing, so too does going back to the utmost primitives in calculating a string of instructions
https://www.hpmuseum.org/rpnvers.htm
The first and only example I can think of is the halting problem. On a practical scale one might prove that the base language doesn’t have memory leaks, so neither can the derived language? What are the advantages of bootstrapping like this? If the answer is simply another form of “because it’s there” (Mallory, re climbing Everest) then I respect that!
It is very neat to see this kind of syntax bootstrapping. I think there's some value (in a researchy-sense) to being able to do that. But I'm not sure if there's something fundamentally "better" about this approach over Racket's approach.
Postscript: Lisp (and Scheme and Racket) macros typically operate on AST (typically because Lisp has reader macros and Racket has a full-bodied reader extension) but Rhombus [3] operates on a "shrubbery", which is like an AST but it defers some parsing decisions until later. This gives macros a little flexibility in extending the syntax of the language. Another interesting point in the design space!
[1]: https://docs.racket-lang.org/guide/hash-reader.html
[2]: https://docs.racket-lang.org/datalog/datalog.html
[3]: Flatt, Allred & Angle et al. (2023-10-16) Rhombus: A New Spin on Macros without All the Parentheses, Proceedings of the ACM on Programming Languages. https://doi.org/10.1145/3580417
1: Which is powerful enough to implement a C compiler in: https://github.com/vsedach/Vacietis
In Lisp they don't. See Emacs Lisp, Common Lisp, ISLISP. A Lisp macro gets only passed some data and returns some data. There is nothing like an AST.
If we define a macro foo-macro and we call it:
then ... can be any data.Example:
Above macro REV gets data passed and reverses it. The &rest list of ITEMS gets reversed. ITEMS is the list of source arguments in the macro call.We can then write:
Example: There is no syntax tree. All the macro does, is to reverse the list of its source arguments.If I let the macro describe the thing it gets passed, then we get this:
As a side effect, we can see that the macro gets a simple list passed. A list of numbers and a symbol ("symbol" is also a data type). Not text. Not an AST.Also data which is possibly not read, but computed by other code. We can compute the arguments of the macro REV and pass the thing to EVAL. Works the same as above.
The macro only gets that data and can do anything with it. There is no idea of Abstract Syntax Tree: it does not need to be a tree, it does not need to be valid Lisp code and it carries no syntactical information. Generally, it also does not need to return valid Lisp code. To be computed all we need is that the evaluator eventually sees valid Lisp code, not the individual macro.In Lisp the "reader" by default only parses a data layer: symbolic expressions. EVAL, macros and other Lisp functionality gets mostly data passed. EVAL has to figure out, what (a b c) actually is as a program. It can traverse it in an interpreter or compile it. A compiler may internally create AST representations -> but that free to the implementation.
The Lisp language then typically is not defined over text syntax, but data syntax.
A Lisp interpreter processes during execution not text, but s-expressions. It's a "List Processor", not a Text Processor. Lisp not Texp. ;-) The function COMPILE gets an s-expression passed, not text.
Racket and Scheme have other macro systems.
For example I can make a circular list being a part of the source and the macro may process it.
Get the first two items from a list and add them:
ADD-2Now we construct a circular list and use the macro:
The circular list is really a part of the source code: Another example: One could pass in a string and have the macro parse the string...Scheme did implement actual syntax objects and other lisps may have a concrete ast (pun slightly intended) layer but it's not required to enjoy the benefits of sexp as code and data.
my two cents
An AST would be the product of a parser and the parse would parse the code above according to some provided syntax. The AST then would encode a tree built from some syntax, where the nodes would be classified. In Lisp symbols and lists have multiple uses and the s-expression does not encode which uses it is: is it data, is it some kind of syntactical element. There is also no defined parser and no syntax it encodes on that level. READ is at best an s-expression reader, where it has zero knowledge what the lists and symbols supposed to mean in Lisp. For the reader (+ a b), (a + b), (a b +) are just s-expressions, not Lisp code.
Where there serious defects caused due to this dynamic nature (honest question) ? it seems to me that people adjusted to this without big troubles. Not that I'm against any improvement.
Nanopass shows an example of using "rewriting" while keeping to pretty much same structure for AST till you end up with native code.
The compiler will expand the macros at compile time and the generated source code then is checked.
One thing this macro system enables are macros which can implement relatively arbitrary syntactical extensions, not restricted by a particular syntax. That can be seen as useful&flexibility or as potential problem (-> needs knowledge and discipline while implementing macros, with the goal of ensuring the maintainability of the code).
In the lisp world, when data represents code, it's probably stored in a tree. Elsewhere, when data represents code, it's probably stored as an array of bytes. There code may not be recognised as being data at all.
It's the difference between eval taking a byte string and taking a parse tree. Rather literally.
This is only a convention. Lisp does it right, but as almost everything else thinks code is strings, confusion abounds.
The data AST is rich enough to for ergonomic, precise source-to-source manipulation. Even a pretty advanced Lisp compiler can be built which goes straight from the data AST to an intermediate representation, skipping the code AST stage.
"Data AST" means that when we have (+ 1 2), this doesn't say "I'm an arithmetic expression", but rather something weaker: "I'm a list of three elements: a symbol object, and two integer objects".
The list object is an abstract syntax tree for the printed list. It must be. It not a parse tree because a parse tree would preserve the representation of the parentheses: in a parse tree, every grammar symbol appears: the nodes of the tree are 1:1 to the grammar rules that they match. Since the parentheses, and whatnot, are gone, it must be abstract syntax.
Most of Lisp syntax is designed such that its syntactic units correspond to nodes of the data AST. That makes source-to-source transformations ergonomic, because the data AST data structure is easy to manipulate.
It's inspired me to mess around in this area again, and I'll definitely have to re-read your blog post a few times.
STOPPARSE
It would be defined as causing the normal parser to stop, and any text / bytes after that keyword is fed as input to the program.
__DATA__ sooort of counts and is probably the closest to pure STOPPARSE as GP is thinking of.
I also, once, wrote https://p3rl.org/Devel::Declare which basically grabbed the current read position and Did Things, including calling back into the existing perl parser to not have to re-implement everything - that was how perlers first got access to a mostly* working 'method' keyword.
(now there's a proper way of doing that in perl core, and Devel::Declare is (very happily to me) obsolete, but it was essential to prove the concept and the user interest to justify adding the actual feature to the core interpreter)
[*] the caveats are many and believe me I'm well aware the entire thing was a giant hack from the moment I first started writing it, but rather than trying to write them out I'll just say "however hacky you imagine this was, what I actually did is probably worse" and leave it there.
That just gives you arbitrary input. You can write your own interpreter for a language and do the input in that language, but that’s not “arbitrary metaprogramming” in the host language, at least as most people understand it. Or you can leverage whatever metaprogramming facilities are offered by the host via an external DSL you implement, which may be a more ergonomic way of metaprogramming the host language, but still is limited to the metaprogramming facilities exposed by the host language, so it’s only “arbitrary metaprogramming” to the extent that the host language already exposes arbitrary metaprogramming.
Doesn't that sound like TeX \catcode? In fact, I think you can bootstrap Cognition from TeX as well.
There's clearly something deep going on, but I will have to come back to this after an even deeper cup of coffee.
I suspect the reception of it being suspected to be a joke, is the mention of lisp and brainfuck priming them for a joke, combined with examples and concepts that seem to require a much stronger than normal technical background. So for the average Joe it ends up in the “turbo encabulator” zone where it’s not quite clear if what’s going on is real or in jest. The prerequisites for understanding just aren’t there.
I also suspect a non-zero percentage of the readers have involuntarily audio flashbacks of Soulja Boy when they see that many “cranks” on a page.
Your post sounds like an interesting research project, but the examples given in the blog post seem somewhat discouraging to me. When I gaze at one of your examples, they seem obtuse and while I am sure I could understand the examples by not just skimming the article, this does tell me that the grammars you used are not very self-explanatory: Your syntax expresses meaning relative to the specific grammar as opposed to expressing meaning relative to the English language or pre-existing, well-established programming languages.
Is there some example of self-explanatory grammars and self-explanatory bootstrapping code in your language?