Postgres Language Server (github.com)
there have been previous attempts at adding Postgres support to code editors. usually these attempts implement a generic SQL parser and then offer various "flavours" of SQL.
This attempt is different because it uses the actual Postgres parser to do the heavy-lifting. This is done via libg_query, an excellent C library for accessing the PostgreSQL parser outside of the server. We feel this is a better approach because it gives developers 100% confidence in the parser, and it allows us to keep up with the rapid development of Postgres.
this is still in early development, and mostly useful for testers/collaborators. the majority of work is still ahead, but we've verified that the approach works. we're making it public now so that we can develop it in the open with input from the community.
a lot of the credit belongs to pganalyze[1] for their work on libpg_query, and to psteinroe (https://github.com/psteinroe) who the creator and maintainer.
[0] LSP: https://microsoft.github.io/language-server-protocol/
[1] pganalyze: https://pganalyze.com/
125 comments
[ 3.2 ms ] story [ 177 ms ] threadit's a great idea though - I will add it to the "Discussions" to track for the future. I can personally see the benefit of PostGIS support
https://github.com/Borvik/vscode-postgres
I am regularly surprised how bad the tooling is for SQL/databases. Especially that there is no decent formatter that supports plpgsql, and doesn't absolutely mangle your queries (sometimes actually breaking your queries..).
Best options atm are imo TablePlus and DataGrip, I have tried a bunch of options in vscode and although there are useful tools, it's all not really there.
(Also excited to see what else they will release this launch week!)
Thanks for the mention!
Lol, this is too true. I use Beekeeper Studio every day. Every once in a while I think to myself, "I should see if the full version is worth paying for yet". Then I check out the features and, while they look great, the free version has absolutely everything I need.
Thanks for such a great product. I'll stop freeloading eventually!
Anyways, once taking the time to re-read the pricing screen subtleties it is sort of a one-time fee (billed as one year of subscription to gain indefinite use), so kudos to the developer, but I have to say it's really non-obvious to me at least.
Also the availability of a community edition is great, but again non-obvious.
The difference is - whether it’s a purchase or a subscription.
I’ll happily buy a tool, and happily will buy upgrades every so often.
I will absolutely resist paying for subscriptions unless you are delivering some sort of continuous service and I won’t be fooled by gimmicky cloud integrations just to make that claim.
I’m ok with the model of TablePlus however: pay for a license, it’s valid for a year, and at the end of the year you keep the current version available forever but no more updates. And you get a discount if you decide to renew your license
I’ve been a happy subscriber to Jetbrains Toolbox that has a similar model for many years now, and have interrupted and later resumed my subscription while being able to use the software in between.
100% against a subscription where you no longer able to use the software the moment you cancel your subscription after paying for years. Looking at you Adobe.
It looks quite similar to table plus, any killer features I’m overlooking?
What I really want is a state-based way to define my database. EdgeDB & Prism have made some progress here but they really only handle table changes. Really wish this were built into Postgres somehow (or any solid migration story).
I want to be able to define my tables, views, functions and everything and the engine behind it automatically figures out all the necessary changes. For example, change a table (add a column or something) and it is smart enough to update any dependent views and functions. Including handling of data migrations.
There are other notable mentions in this space:
Reshape: https://fabianlindfors.se/blog/schema-migrations-in-postgres...
Atlas: https://atlasgo.io/
A "custom" example on GitLab: https://gitlab.com/delibrium/delibrium-postgrest/-/tree/mast...
[0] https://github.com/kiwicopple/declarative-schemas
I don’t like the solution of “stop using views and functions” as I prefer to use use the full power of Postgres (everything close to the data, less application level logic, great plugins). Not sure what approach should be taken to creat a state based migration solution, but it feels like the tech around Postgres, language servers, MML, etc has matured enough to solve the issue in an automated way.
Not as advanced as you describe though, it won’t handle your data migrations.
Agreed. Even when using something open like SQLite, you will find a lot of difficulty in answering simple (to me) questions like "how do I parse this command text into an AST".
Tooling for Postgres seems to be more for the technical who prefer command line than not.
At this point my muscle memory is on the Sequel Pro shortcut keys, except I'm finding myself around Postgres more and more.
This. I'd really like to have a working SQL pretty printer/formatter. Psql is great, but lacks autocompletion in several places.
That said, a big shoutout to pspg (https://github.com/okbob/pspg) - an excellent pager for psql (also for general csvs).
(ETA: Being a big fan of Merge statements for a bunch of ETL trivia is partly how I got to "Sentence case". Merge statements are tough to read at the best of times and eliminating extraneous casing changes as noise by using more lower-case than not was a very useful thing to me. I don't need to highlight keywords versus everything else in a modern syntax highlighting bath, so casing becomes useful for way finding. A lower-case "select" is in the middle of something else.)
Is it that I type in my IDE "SEL" and is suggests "SELECT"?
In which situation would this be useful?
Different editors and IDEs can use the same LSP, without the need to reinvent/reimplement the wheel.
I have a couple of questions:
1. Will I be able to use this in my IDE (say JetBrains or VSCode) alongside the programming language’s LSP? So that this project would only be in effect for the Pg SQL bits?
2. I am a Rust newbie but I’m an experienced developer. Is there any room for someone like me to contribute to the project?
2. Yes, absolutely. Feel free to reach out to me directly, or just start contributing directly if you want
I think that would depend on IDE support. For example, JetBrains IDEs have a feature for this.
> Starting with the 2023.2 release cycle, the LSP API is publicly available as part of the IntelliJ Platform in the following IDEs: IntelliJ IDEA Ultimate, WebStorm, PhpStorm, PyCharm Professional, DataSpell, RubyMine, CLion, Aqua, DataGrip, GoLand, Rider
It checks the syntax, gives you autocomplete based on the databases you are connected to, it even supports string language injection, so you'll get SQL autocomplete and checking for strings containing SQL in your code.
I'd love to understand what the advantages here are, other than the obvious one of being available for many editors as a language server.
You wouldn't really notice a difference using this, except your general autocompletes would get worse, and very complex sql might get more correct (I break datagrip's parser occasionally)
As someone interested in developing a client for it, I'm interested in couple of things: what are the features supported by it currently, the tweak-able configuration that can be passed to it and the various code action available. I like the way nil language server has documented it (https://github.com/oxalica/nil/tree/main/docs). Is there something equivalent available for this ?
your comment is useful and why we submitted this to ShowHN - it's easier to get these sort of early feature requests than in 3 months once we've implemented more functionality. I'll drop a note in GitHub Discussions to investigate the Nil approach.
https://github.com/mason-org/mason-registry/
"Optionated"? Opinionated or optional? Or both?
I've updated the Readme
I like the focus on this.
we talked in the past about using their library to extend some of Dan Lynch's work (https://github.com/pyramation/postgres-ast-deparser) to create a sort of "postgres package registry" with dynamic DDL introspection. No progress on that one yet, but I'm hopeful that we'll find the bandwidth for it one day.
I'm thankful to Supabase for their efforts, but I'm not sure I'd pick pgAdmin and migra as examples of "really effective" collaboration with other teams: as far as I know, pgAdmin's CLI feature wasn't merged upstream and sits in an outdated Supabase fork of pgAdmin which doesn't fully support Postgres 15, and migra (which has been abandoned for a while) was just used as-is.
(but I agree with the sentiment: we can always find ways to be more collaborative)
At Omnigres [1] we also use libpg_query for programmatic query manipulations as libpg_query has a deparser and that allows one to manipulate AST and output the text representation back.
I'm trying to find time to contribute Postgres 16 support to it!
[1] Omnigres: https://github.com/omnigres/omnigres
We at pganalyze have plans to add Postgres 16 support to libpg_query sometime between the RC1 and the final release - mostly to watch out for the rare case where a feature gets reverted during the beta releases.
If you'd like to have a go at it before then definitely welcome (here is some context on the extraction process using libclang: https://pganalyze.com/blog/pg-query-2-0-postgres-query-parse...), and we'll also need people to test the PR once available.
I'd recommend starting with fixing that instead. It would be much more helpful for the community.
fwiw, we know of this limitation and decided the acceptable workaround for Windows devs is simply to host the language server and they can connect to it via TCP.
It seems that the only problem for fixing the problem is:
> Thanks for the offer, but the problem is our team being time limited / having an engineer with a Windows machine ready to take this on, not that we wouldn't want to pay someone to work on it :)
(https://github.com/pganalyze/libpg_query/issues/44#issuecomm...)
Hosting the LSP elsewhere is not really needed since if people wanted to go that way they could use Remote ssh (https://code.visualstudio.com/docs/remote/ssh) to host the whole dev environment on linux and connect to it.
Thank you
(Speaking as someone who dreaded the “inevitable” victory of Windows on the desktop in the 90s I might be a little biased/overjoyed.)
> the only problem for fixing the problem is
Hopefully it’s as easy as you make it sound!
;)
Postgres itself supports Windows, the main challenge is identifying the relevant files to pull in (since we use a subset of the full source code), and testing the changes. It should be fairly quick for someone who has C development experience on Windows, with a working development environment.
[1] https://sql.ophir.dev
I feel that this approach, while unfashionable, is actually pretty excellent most of the time.
(I'm a vim user but I'll accept a VS Code answer since that's basically "state of the art" when it comes to LSP and where most others get their inspiration from.)
-- Initialization of Neovim's LSP client local nvim_lsp = require('lspconfig')
-- Setup for the Ruby LSP nvim_lsp.solargraph.setup {}
-- Setup for an SQL LSP (assuming one exists with the name 'sql_lsp' in this example) nvim_lsp.sql_lsp.setup {}
This should be configured by the user, idk if a default implementation could work correctly in all cases.
This seems to have been a design decision from the very start, but I have a hard time understanding why. It seems like a conceptual mistake to me. It also eats up system resources like crazy, but that's a lost cause anyway. Instead of harnessing the capabilities of existing implementations, every "wrapper" kind of language needs to build its own solution, which leads to these solutions often not being great. For example, said HTML/JS solution has issues with external dependencies [3]. Or, I've been trying to enable support for CoffeeScript or Elm inside Vue Single File Component files, but I couldn't so far, even though I'm the author of the CoffeeScript extension. [4][5]
Any source file always has exactly one language type assigned to it (possibly auto-detected), so no, dynamically "switching" is not possible without manually changing it for the entire file. For things like SQL inside PHP, Jetbrains products really are superior.
[1] https://code.visualstudio.com/api/language-extensions/embedd... [2] https://github.com/microsoft/vscode/blob/7fd6f1b1d46bd2223a9... [3] https://github.com/microsoft/vscode/issues/26338 [4] https://github.com/vuejs/language-tools/issues/3200 [5] https://github.com/vuejs/vetur/issues/1184
Less sure about "minor" dsl-like support, like online css/html/js in php etc? (Or SQL strings in c# etc).
If anyone else has a use case for using the Postgres parser outside the server, we have a healthy ecosystem of libraries that build on the core C library (we maintain bindings for Ruby, Go and Rust ourselves), as well as various projects using it (e.g. sqlc uses it for a type-safe way for using hand-written SQL in Go): https://github.com/pganalyze/libpg_query#resources
I imagine that Postgres itself would then consume the parser library as a — possibly regularly-snapshot-vendored — static library dependency; while other applications and wrapper libraries would be free to consume it as a dynamic shared library; and it could be freely independently distro-packaged; and so forth. In other words, it would become a de-facto "libxml for SQL" — the kind of venerable C lib that you expect everything else to just be a wrapper around.
Is it just that doing this would involve making functions with internal linkage into functions with external linkage, and thereby preventing some WPO opportunities when compiling Postgres itself?
However, the challenge from what I've understood from past conversations with some folks working on Postgres core is that the parser is currently heavily tied into the backend - note the parser isn't just the scan.l/gram.y file, but also the raw parse node structs that it outputs. You can see how many files we pull in from the main tree that are prefixed with "src_backend": https://github.com/pganalyze/libpg_query/tree/15-latest/src/...
Further, there isn't a canonical way to output node trees into a text format today in core, besides the rather hard to work with output of debug_print_parse - there have been discussions on -hackers to potentially utilize JSON here, which may make this a bit easier. Note that in libpg_query we currently use Protobuf (but used to use JSON), which does have the benefit of getting auto-generated structs in the language bindings - but Protobuf is not used in core Postgres at all today.
All in all, I think there is some upstream interest, but its not clear that this is a good idea from a maintainability perspective.
My heroes, thank you for working on this.