Another issue with tree-sitter is how basic functionality is not consistent across all languages. Want to find all comment text in all languages? Each tree-sitter lang support will have a slightly different behavior for every language. When you extract the comment you'll have syntax from the language in there like `/*/` or `{}`.
Example from a sourcegraph docs generation project:
Regarding the `foo() { bar() {` example, I wonder if a possible approach would be to determine in which syntactic contexts the local token sequence around the parse error (here: `bar() {`) would be correct, and then try to determine what is missing for that context (here: class scope) to have been established; or just give the error “method definition in local scope”, but still parse it as a method definition.
As it currently works, the interpretation as a method definition is not even considered, because the grammar doesn’t have method definitions in that syntactic context. But that’s not how human pattern recognition works: we tend to recognize a method definition regardless of the context. So maybe we could build a parser that would try do determine all possible valid interpretations of all token subsequences in parallel (like a nondeterministic generalized parser), outputting errors where neighboring /overlapping sequences clash which each other (“this starts out like a method body but then transitions into a method definition”).
2 comments
[ 3.2 ms ] story [ 11.5 ms ] threadExample from a sourcegraph docs generation project:
https://github.com/sourcegraph/doctree/blob/07a2d593fe9855be...
https://github.com/sourcegraph/doctree/blob/07a2d593fe9855be...
As it currently works, the interpretation as a method definition is not even considered, because the grammar doesn’t have method definitions in that syntactic context. But that’s not how human pattern recognition works: we tend to recognize a method definition regardless of the context. So maybe we could build a parser that would try do determine all possible valid interpretations of all token subsequences in parallel (like a nondeterministic generalized parser), outputting errors where neighboring /overlapping sequences clash which each other (“this starts out like a method body but then transitions into a method definition”).