Postgres.new: In-browser Postgres with an AI interface (supabase.com)
This is a new service that we're experimenting with that uses PGLite[0], a WASM build of Postgres that runs in the browser. You might remember an earlier WASM build[1] that was around ~30MB. The Electric team [2] have gone one step further and created a complete build of Postgres that’s under 3MB.
Their implementation is technically interesting. Postgres is normally multi-process - each client connection is handed to a child process by the postmaster process. In WASM there’s limited/no support for process forking and threads. Fortunately, Postgres has a relatively unknown built-in “single user mode” [3] primarily designed for bootstrapping a new database and disaster recovery. Single-user mode only supports a minimal cancel REPL, so PGlite adds wire-protocol support which enables parametrised queries etc.
We have created https://postgres.new as an experiment. You can think of it like a love-child between Postgres and ChatGPT: in-browser Postgres sandbox with AI assistance. You can spin up as many new Postgres databases as you want because they all live inside your browser. We pair PGlite with an LLM (currently GPT-4o) and give it full reign over the database with unrestricted permissions. This is an important detail - giving an LLM full autonomy means that it can run multiple operations back-to-back: any SQL errors from Postgres are fed back to the language model so that it can have a few more attempts to solve the problem. Since it’s in-browser it’s low risk.
Some other features include:
- CSV upload: you can upload a CSV and it will automatically create a Postgres table which you can query with natural language.
- Charts: you can ask the LLM to create a chart with the data and change the colors of the charts.
- RAG / pgvector: PGLite supports pgvector, so you can ask the LLM to create embeddings for RAG. The site uses transformers.js [4] to create embeddings inside the browser.
Everything is open source. A huge shout-out to the Electric team who have been a pleasure to build with.
[0] PGLite: https://github.com/electric-sql/pglite
[1] Postgres-wasm: https://supabase.com/blog/postgres-wasm
[2] Electric: https://electric-sql.com/
[3] Single user mode: https://www.postgresql.org/docs/current/app-postgres.html#AP...
[4] transformers.js: https://github.com/xenova/transformers.js
[5] pg-gateway: https://github.com/supabase-community/pg-gateway
115 comments
[ 4.5 ms ] story [ 180 ms ] threadWhat the team at Supabase have built with postgres.new is incredible, it has been a lot of fun to work with them on it over the last month or so. It's fead directly back into the development of PGlite and helped us iron out some bugs and finish hitting the feature list required.
SQL databases and LLMs go together so incredible well, the structured data and schema enables an LLM to infer so much context about the database and the underlying data. This exploration into a UI over this paring is going to be incredibly influential, it opens up what has traditionally been a complex technical problem to everyone. That's the true power of LLMs.
I'm not going to go on about PGlite here as postgres.new really deserves the limelight!
also is the UI tool open source? id prefer to run it locally
edit: this is an incredible tool. its going to eat a lot of backend engineers billing hours.
suggestion: add a copy button for the generated SQLs
* postgres-new (https://github.com/supabase-community/postgres-new): The frontend. Apache 2.0
* PGlite (https://github.com/electric-sql/pglite): A WASM build of Postgres. Apache 2.0
* pg-gateway (https://github.com/supabase-community/pg-gateway): Postgres wire protocol for the server-side. MIT
* transformers.js (https://github.com/xenova/transformers.js): Run Transformers directly in your browser
Really impressive stuff - congratulations!
Edit: Okay reading kiwicopple's comment makes it clearer that chatGPT is not-optional. I'm... not enthused by this. Why would you take something that's local-first and gatekeep it with something as useless as an AI-for-your-db?
https://pglite.dev/repl/ is available as a more barebones browser-pglite playground.
That said: it's 100% our intention to add local models. This is just our v1.
Also... the page title is "Postgres Sandbox". This is, at best, misleading.
we'll change the verbiage for now, and look at ways we can provide a 100% local experience without logins
Then you want pglite. This project uses it and provides a link to it.
pglite.dev/repl does not have the same level of visualizations as postgres.new.
What I 'want' is exactly what I described in my previous message, postgres.new but without the required LLM integration, the same sentiment others had in this thread.
https://wasm.supabase.com/
Be aware that the WASM file is 30Mb, but i think it fits what you're describing. details here: https://supabase.com/blog/postgres-wasm
Open Weight models specifically fine-tuned on sql generation and modification also rank pretty well compared to SOTA proprietary models. If you want to eval alternative models, check out sqleval [2]
1 https://prollm.toqan.ai/leaderboard/stack-unseen?type=concep...
2 https://github.com/defog-ai/sql-eval
You need to start typing after clicking on the button. It needs some UX rethinking, but no bugs, just start typing on the chat
(I still hate LLMs in my DB. I KNOW SQL LET ME WRITE IT.)
My biggest prediction is that ORMs will not be necessary when LLMs can generate SQL. Low level SQL is a better abstraction for LLMs than ORMs, and as people are removed from the equation, so too will abstractions built to help them craft code.
Why wouldn’t the LLM use an ORM?
Not only that, but AI is way more expensive than we think. We're currently in a hype bubble funded by last ditch effort VC money. When that money runs out, and it will eventually, AI is going to get WAY more expensive.
An LLM is just taking prefabbed templates and swapping the possibilities in the answer for a statistically relevant solution. My code generator outputs a prefabbed template with a deterministic solution no statistical guessing required.
Let’s say there is a reason to keep this 10K LOC together in a single file. I have never had work in SWE that involved making minor iterations to a file over a week where the work took a whole day to complete. I can see how that could happen but requiring a day to change code seems like there are bigger issues than a 10K LOC file. Unless I’m a complete amateur that thinks they’ve always been not an amateur, and needs an LLM to make a minor change. I just don’t see the point a lot of the times.
What do I do with all this extra time I’m saving? Retire early because I’m getting paid more for doing less right (and I saved all that time)?
How about when the LLM doesnt work right? If I’m a junior engineer that lets a computer write everything for me; How much time do I spend hacking at a prompt to get what I want vs just writing the damn code?
The utility of LLMs with code generation varies widely with the problem domain and the amount of experience the developer has.
I fear applications will suffer the same fate. "Good enough" will take over "well-architected".
Couple this with decreasing costs of storage (and ideally compute), and it doesn't matter if the data model is garbage, people can still get something workable that's better than the awful Excel files they curate now. It will still make errors, but eventually fewer than their spreadsheets.
There is no "good enough" for data modeling. There is correct, and there is "this works, but it has latent bugs that will eventually surface." You either have referential integrity, or you don't.
In the 20th century you got away with knowing the syntax and hacking away. Now you really need to have a deep understanding of relational algebra, since the LLM is doing the typing for you.
Until a query becomes a bottleneck and no one knows why because no one knows how databases work anymore.
2 hours ago | 41 comments: https://news.ycombinator.com/item?id=41224689
You'll just provide an openAI/anthropic api key and connection details to the db/schema. I intend for it to work a lot like postgres.new but with regular postgres instances.
https://news.ycombinator.com/item?id=41227656
My implentation is more focused on data analysis and visualization via a natural language interface.However, the straighforward database operations that postgres.new tries to tackle are included.
https://aws.amazon.com/blogs/machine-learning/build-a-robust...
There's nothing wrong using Webkit / Safari on your laptop or desktop. There are dozens of us, DOZENS!
It throws an error when its full screen though. There is more to the error, but it wont let me copy it.
1/ you're using a small window (even on Chrome/FF). We don't support mobile yet, so if you're seeing this on a desktop just expand the window
2/ you're on safari, which doesn't have support for OPFS yet: https://github.com/electric-sql/pglite/pull/130
How will it perform if I have 1TB of data?
PGlite is working on an OPFS backend, which will likely increase the max data size quite a bit.
Good work
- a customer orders database with products with a timeseries of prices, and multiple fulfilments per order. - an issue tracking system with a reflexive user/manager database - a family relationship model
In each case I got it to put in sample model and then asked postgres.new to answer some questions about the data it had inserted. I thought the family model would trip it up, especially when I told it to put in cousins and uncles. But no, it was pretty bang on.
The only thing it didn't quite manage is that some of the relationships are reciprocal (i.e. my sibling also has me as a sibling).
I asked postgres.new to review the data and it fixed some, and i asked it to check again and it fixed the rest. This is a very useful tool that I can see myself using!
I can see some situations you might want direct access to postgres, so we'll jot that done as something to look at too
I'll flag this with the team but we can't actually access your "db id oy6g6g6o5qbt78fo" as it's run directly in your browser and doesn't touch our servers at all
Adding offline support (PWA) to an application which was primarily designed for use on a server. Though I'm not sure how well that'll work in practice, since the rest of the application will be designed to run on a server as well.
The Electric team have built this extension which will fit that use-case very well: https://pglite.dev/extensions/#live
Overall, looks pretty good. I’m on mobile but stumbled upon the blog post in comments.
Ask: I understand why it won’t run on mobile but at least give mobile a synopsis of what it’s suppose to do. I almost ignored this if it wasn’t for the luck of seeing your comment.
agreed - we rushed this one a bit. we're working on some some updates now for mobile users
edit: we've shipped some changes for mobile which embed the video and link to the blog post so that it's a bit clearer. Thanks for the feedback
infinite error loop
In the clause, "you" refers to Supabase. This is a Terms of Use, not a copyright license. ChatGPT text is not copyright protected and the protection of that clause does not extend to the text or to downstream consumers of the text.
The clause is entirely to prevent people asking ChatGPT to generate training data they feed into another model.
Also, does the WASM build perhaps enable using Postgres as an embedded db, where typically SQlite would be used?