Show HN: GritQL, a Rust CLI for rewriting source code (github.com)
I’m excited to open source GritQL, a Rust CLI for searching and transforming source code.
GritQL comes from my experiences with conducting large scale refactors and migrations.
Usually, I would start exploring a codebase with grep. This is easy to start with, but most migrations end up accumulating additional requirements like ensuring the right packages are imported and excluding cases which don’t have a viable migration path.
Eventually, to build a complex migration, I usually ended up having to write a full codemod program with a tool like jscodeshift. This comes with its own problems:
- Most of the exploratory work has to be abandoned as you figure out how to represent your original regex search as an AST. - Reading/writing a codemod requires mentally translating from AST names back to what source code actually looks like. - Performance is often an afterthought, so iterating on a large codemod can be painfully slow. - Codemod frameworks are language-specific, so if you’re hopping between multiple languages—or trying to migrate a shared API—you have to learn different tools.
GritQL is an attempt to develop a powerful middle ground: - Exploratory analysis is easy: just put a code snippet in backticks and use $metavariables for placeholders. - Incrementally add complexity by introducing side conditions with where clauses. - Reuse named patterns to avoid rebuilding queries, and use shared patterns from our standard library for common tasks like ensuring modules are imported. - Iterate on large codebases quickly: we use Rust for maximum performance
GritQL has already been used on thousands of repositories for complex migrations[1] but we're excited to collaborate more with the open source community.
[1] Ex. https://github.com/openai/openai-python/discussions/742
57 comments
[ 5.0 ms ] story [ 87.6 ms ] threadManaging tech debt is and will be an important issue in the world of AI software. I'm glad there's a company solving this. Additionally, this is something the current generation of LLMs are already helpful at solving. GritQL is very unique way of approaching this problem.
OT: We've been working with Grit and Morgante for the past months and I have only positive things to say about them. If GritQL looks useful to you, you should reach out.
We've tackled some pretty gnarly migrations with GritQL and it has saved us a lot of time.
We have used ts-morph in the past for similar types of migrations but found the barrier to entry felt high, which prevented adoption. We were impressed by how complicated of migrations we could use GritQL for—we have been using it to migrate from mobx to react hooks.
Your suggestion to add the `--interactive` flag was hugely helpful, it's become very commonly used.
Interested folks may also want to check out ast-grep:
https://github.com/ast-grep/ast-grep
The main area GritQL shines is in taking simpler transformation patterns and composing them into complex migrations. For example, the OpenAI migration was built by incrementally handling the different edge cases we've seen in the wild: https://github.com/getgrit/stdlib/blob/main/.grit/patterns/p...
I've spent the last 6 months building a static analyzer that mostly uses ASTs and couldn't understand what you were offering after skimming the readme until I saw this link.
Now I'm thinking of how I could integrate something like this into my analyzer to simplify the implementation of some of the simpler rules.
Let me know if I can be helpful with integrating GritQL.
Not that I'm complaining ;)
Is it a new way to market a C tool?
there are very similar complaints about "yikes that thing just vomits a massive amount of loosey goosey C" so I hope you've not experienced the crashes they're talking about
Do you envision going up against CodeQL and/or <https://www.jetbrains.com/help/qodana/about-qodana.html> by making semantic information available to the ast nodes? OT1H, I can imagine it could be an overwhelming increase in project scope, but OTOH it could also truly lead to some stunning transformation patterns
e.g. https://github.com/github/codeql/blob/v1.27.0/java/ql/exampl... or even more "textual" semantics such as
Since you already have GitLab on your docs page, if I could pay for CodeQL over on GL that would be a game changer IMHOWe started with the AST, since that's ultimately the foundation for code but the goal is to add more graphs on top. So you could do a query like this to find all cases where a function called `unsafe` is queried with an unknown value:
`unsafe($val)` where $val <: type `unknown`
Is your interest in CodeQL primarily for security scans?
I recalled another link I wish I had included in my question from the SourceGraph folks https://github.com/sourcegraph/scip#scip-code-intelligence-p... which started out life as "Language Server Indexing Protocol" and seems to solve some similar project-wide introspection questions but TBH since their rug pull I've been a lot less willing to hitch my wagon to their train
We've looked at both LSIF and SCIP from SourceGraph, though I expect we'll end up building our own index to allow for more complex queries. We're also incorporating some LLM functionality to express conditions you can't program deterministically.
If you're interested in being a design partner for some of our auto-review features feel free to email morgante@grit.io.
GritQL shares a lot of common ground with ast-grep, which is also a code rewriter based on tree-sitter.
Interested folks may wonder about their difference. The key difference is that their surface APIs are quite different. GritQL is more like a DSL (or SQL). while ast-grep is more like pattern language + embedded configuration in YAML. Check it out here if you are interested https://github.com/ast-grep/ast-grep
I'm just starting to learn about tree-sitter but my understanding is that it can also handle nested syntax, for example code in documentation html pages. Does sg support this a well?
What I would like to do is use sg to update my project's code samples. Is that possible? What part of the docs should I look at for this?
For update code, I recommend first read the quick start, pattern and rule essentials
https://ast-grep.github.io/guide/quick-start.html
https://ast-grep.github.io/guide/pattern-syntax.html
https://ast-grep.github.io/guide/rule-config.html
Specifically,
Where $conn needed to be of a specific connection class to be a match.I ended up taking a Java parser, adding simple type resolution myself, and then the rest of the find-replace logic from there. I wouldn’t expect a tree-sitter based tool to deal with types, but if there’s a hook to add metadata to the parse tree and replace utilizing it, then this might have saved me some trouble
In practice, it's not really the ideal tool for translating between languages.
It does work well for cases where the differences are minor (ex. converting from JavaScript to TypeScript, or between different SQL dialects).
In your opinion, what tools would be ideal for general purpose transpilation between languages?
Yeah, that should be doable (though GritQL doesn't have PHP support yet).
> In your opinion, what tools would be ideal for general purpose transpilation between languages?
Writing a transpiler is quite complicated and requires a lot of development to get right. I don't know of any high-quality generalized transpilers—you're best looking for tools specific to the language pair you're targeting.
Does the rewrite code also have to be syntactically correct or is that a simple text replacement because that's what I see as the main impediment; if I'm using the sqlite grammar to parse and produce postgresql output, the output won't conform to the grammar.
Also how do I dynamically load custom tree-sitter grammars like in ast-grep?
No, we don't require the output code to be syntactically correct. You can use the raw`code` modified to indicate a raw text replacement: https://docs.grit.io/language/patterns#raw-output
> Also how do I dynamically load custom tree-sitter grammars like in ast-grep?
We don't currently support dynamically loading grammars, mainly because our metavariable approach requires some modifications to the grammar.
A simple application would be to produce a code formatter like Black or ruff for python but which could support any tree-sitter supported language.
More specifically to my use case though, if I produced tree-sitter grammars with identical node structure and node names, e.g. for sqlite and postgresql, then wouldn't this make it simple to do full transpilation between the two?
Yes, theoretically if you had ~identical grammars you could use it to do a full transpilation. There's a lot of challenges with that though. Writing a correct grammar for 1 language is complicated enough, but writing one for two where all your nodes and fields end up the same is likely insurmountable.
In practice, languages are either:
- Far enough apart that any pure AST transformation is insufficient and you need an AI component to produce usable output
- Close enough that you're better off just targeting the specific parts that differ and rewriting those while leaving the rest alone. I think GritQL can do well here.
The ability to write rewrites as regular code fragments with metavariables feels like some ideas I've had in the past. However, it seems to me that your tool lacks the ability to do something like (in C/C++/etc. terms) "replace every use of S::method($arg1) with S::method2(arg1, nullptr, $arg1)", where you have to do some amount of name resolution to know that x->method is a call to S::method, which seems like it is somewhat less useful for several kinds of rewrites I'm interested in.
(One of the main things I keep playing around with in my head is converting a C codebase into C++ by converting its ad-hoc vtable implementation into regular C++ virtual methods.)
For example: https://app.grit.io/studio?key=TteyWLNGZGWFIC9EOiCRi
`$x.old($arg)` => `$x.new("method", _, $arg)` where { $program <: contains `$x = new Thing($_)` }
---
Long term we want to incorporate more semantic analysis, but for quick rewrites this has been pretty effective for us.
I've experienced how much time these migrations take (and how error prone they are), especially once you have double-digit team sizes, so I definitely see the value proposition.
To expand more on our business model, we see GritQL as a powerful but also relatively low-level tool. It can speed up your migration, but writing the query and generating/testing the PRs still takes considerable time.
If you'd prefer to have migrations handled entirely for you, you can pay us to tackle that work. In additional to providing the migration code, we generate pull requests and hook into your CI pipeline to fix downstream test failures. [0]
[0] https://docs.grit.io/workflows/healing
I'm just starting to learn about tree-sitter but my understanding is that it can also handle nested syntax, for example code in documentation html pages. Does grit support this a well?
What I would like to do is use grit to update my project's code samples with new releases or syntax changes. Is that possible? What part of the docs should I look at for this?
We don't have support for HTML yet, but if you open an issue we're happy to explore it.
One tool I've been looking for (and I wonder if this is such a tool?) is something that can do things like rename variables in functions to "more sensible names". Now, it's not hard to get ChatGPT to read a function and suggest new names for variables, but if you ask it to also do the substitutions, there is an annoyingly high chance it break the function in the process.
Worth trying out though. Perhaps rule creation can be turned into function calls so it wouldn't need to understand new syntax.
Applications
1. minification. 2. cross programming language code generation. 3. Source tracing complements control flow tracing.
Bringing in AST based solution will complicate the problem. What do you think ?