I noticed that the Java @tokens definition doesn’t seem to support Unicode Java identifiers and Unicode escapes. Since the latter can appear anywhere in Java source code [0], it’s likely impossible to have a fully compliant auto-generated Java lexer/parser from that tool.
I was wondering why they didn't use tree-sitter since this sounds very similar to tree-sitter, tree-sitter is written in C (and thus hopefully faster than a similar project written in TypeScript), and there's even officially support wasm build steps for if you need to run it in the browser.
The linked blog post [0] answered my question though:
> Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very hefty grammar files because it makes the size/speed trade-off in a different way than a web system would.
3 comments
[ 4.8 ms ] story [ 16.4 ms ] threadIn case anyone else was wondering, this is a LR parser-generator that is supposed to be error-insensitive. Here is a java grammar written in it: https://github.com/lezer-parser/java/blob/main/src/java.gram... which doesn't look terrible.
[0] https://javajee.com/unicode-escapes-in-java
The linked blog post [0] answered my question though:
> Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very hefty grammar files because it makes the size/speed trade-off in a different way than a web system would.
[0] https://marijnhaverbeke.nl/blog/lezer.html