Show HN: DoltgreSQL – Version-Controlled DB, Like Git and PostgreSQL had a baby (github.com)
From the company behind Dolt—the world's first fully versioned database—comes DoltgreSQL, which implements PostgreSQL's variant of SQL.
DoltgreSQL is at a very early stage, and we have quite a lot of work left to do, but we'd love to hear all thoughts and opinions! You can read more in the announcement blog post: https://www.dolthub.com/blog/2023-11-01-announcing-doltgresq...
65 comments
[ 3.4 ms ] story [ 143 ms ] threadDo you have a clear position on which PostgreSQL features not to support? I suppose there are more than just some things that won't make the cut because of the architectural decisions.
While I unnderstand the decision, I'm not sure it's the best way to go about it. If you only emulate a subset of PostgreSQL's syntax and features, few people will be compelled to switch because they might be afraid. For greenfield projects, most people would probably choose the MySQL syntax since it's the default.
I don't think this is about the necessity of running the PostgreSQL binary itself (although your approach already removes extensions which for many people is a downer). It's just that you can't trust an emulated system to be 100% equal in behavior (and people rely on implicit behavior of a system all the time, unfortunately) and that might be already enough for a lot of people to not use it.
Have you guys already encountered some things in the PostgreSQL engine that just behave a bit differently from Dolt's engine? If so, what was your approach to mitigate it?
Edit: just wanted to add: I'm really impressed by your work and I'm looking forward to trying this out. I don't mean to be mean, these are genuine questions I have. Congratulations on the launch.
While I unnderstand the decision, I'm not sure it's the best way to go about it. If you only emulate a subset of PostgreSQL's syntax and features, few people will be compelled to switch because they might be afraid.
Eventually, we'd like to support the entirety of PostgreSQL's feature set, even including features like extensions. Dolt (https://github.com/dolthub/dolt), our first product, is the same to MySQL and DoltgreSQL is to Postgres, and we're taking a no-compromises approach to what we support. That, of course, means that there are a lot of features that need to be implemented, but Dolt is already almost there. For the majority of customers, Dolt has implemented everything they need from MySQL.
I'd definitely recommended checking out how Dolt compares with MySQL to see how we're approaching compatibility. All behavior, implicit and explicit, is something that we aim to model, and any deviations are considered bugs that we need to fix. There are exceptions, but those are only used when we feel it's for good reason (an example being how MySQL handles collation cascading in some circumstances).
> Have you guys already encountered some things in the PostgreSQL engine that just behave a bit differently from Dolt's engine? If so, what was your approach to mitigate it?
With DoltgreSQL, it's at an extremely early stage. We're still working on getting the basic functionality working before we rigorously start testing to make sure that we match PostgreSQL's behavior. However, we can point to our approach with Dolt and MySQL for how we plan to handle DoltgreSQL and PostgreSQL. For every feature we implement, we compare the functionality with what is written in MySQL's documentation as a baseline. From there, we move on to comparing the output across a range of input statements. Sometimes the documentation differs from MySQL's own results, and we then try to find out why that's the case (Configuration? Out of date documentation? Bug? etc.).
We also use external benchmarks to measure our correctness versus MySQL. In one such benchmark, containing around 6 million tests, Dolt recently reached 99.99% compared to MySQL (https://www.dolthub.com/blog/2023-10-11-four-9s-correctness/).
I hope this answered your questions! Let me know if you have any more :)
Some people do performance optimisations based on PostgreSQL's inner working (e.g. trying to force data that isn't read often but not small into toast). How far do your ambitions go? Are you planning on modeling internal behavior like this as well? Do you think putting in an abstraction layer like this between Dolt will hurt performance?
Do you have a time frame (probably not) or a roadmap for Doltgres?
Performance optimizations are tackled a bit differently than correctness ones. For the most part, we'll try to use metrics to find weak points in the execution graph and optimize those, but we won't go so far as to try and model the internal performance behavior. In part because our storage format is so different that we'll have different performance characteristics by necessity.
We don't have a public roadmap for Doltgres just yet, but we're hoping to put one out quite soon! We have a lot of low-hanging roadblocks that we want to take care of before we can get a better look at the overall time frame. We should definitely have one up by the end of the month, but I don't want to commit to a time before that.
Really? I was under the impression that PostgreSQL had already won by then.
The versioning features are an important distinction though. Dolt (production ready, MySQL protocol) and DoltgreSQL (pre-alpha, PostgreSQL protocol) are built specifically to address the lack of versioning support in databases, and gaining these versioning features is as easy as swapping out the database you are using for Dolt and DoltgreSQL (once it's finished). MySQL and PostgreSQL are written using C/C++, while Dolt and DoltgreSQL are using Go, so there is no shared code. The storage format is implemented using prolly trees (https://docs.dolthub.com/architecture/storage-engine/prolly-...), which are based on merkle trees (used by Git and Bitcoin), so there is no overlap with any existing database solutions.
The reason that I am asking is because I have a hard time trusting a newcomer to the database competition. And for purposes off discussion, DoltgreSQL will be a newcomer for the first ten years of its existence. Sharing the underlying storage model with Dolt (still a newcomer, sorry) would greatly increase my confidence in it.
Vitess (Handles Wire Protocol)
go-mysql-server (GMS, Implementation of MySQL featureset)
Dolt GMS interface
Storage Engine
DoltgreSQL, right now, shares the storage engine, interface, and a portion of GMS. This is primarily to get something up and running so that we're able to further iterate on a "working" product (i.e. something we can set up testing for). I envision that GMS will split into three, with a MySQL & PostgreSQL core interacting with a base SQL core. Dolt's GMS interface would then adapt to the base SQL core interface. The storage engine is more-or-less set in stone.
And I can not use it for existing projects, because again extensions, and I surely don't want to findout how your implementation differs from the mainstream postgres...
Regarding any differences from mainstream Postgres, you can look to how we've handled Dolt, which is production-ready. It targets MySQL, just as DoltgreSQL targets PostgreSQL, and it recently achieved 99.99% correctness according to a set of roughly 6 million tests (https://www.dolthub.com/blog/2023-10-11-four-9s-correctness/). This test is not a definitive stance that we are exactly 99.99% the same as MySQL, but it's a good general guide to how we approach our compatibility, and how serious we are in that regard.
Sadly though, the full versioning capabilities would not work as an extension to Postgres. We looked into it before we settled on our current approach. I talk a bit about it in the blog post as well. To truly allow versioning in the same capacity that Git does for source code, it required us to either fork Postgres and spend years reimplementing all of the work that we've done in Dolt just to get to where we are today, or choose the path that gets something out quickly, and allows us to have the very conversation that we're having right now.
That was actually the very reason for deciding to host an announcement that we're working on it. Many people have said that they'd like Dolt but for Postgres, however they've not said whether they need the Postgres binary specifically, the ecosystem, the syntax/wire protocol, etc. This announcement gives us the opportunity to receive that feedback.
And most importantly, how does Dolt compare under heavy load, at the limits of server memory or bandwidth or CPU or disk thoroughput? You can assume an SSD and a multicore processor for purposes of answering.
Thank you very much. You are competing in a field where trust is extremely difficult to acquire - and the consequences to a lead dev for choosing Dolt[greSQL] could end his career. Nobody ever got fired for choosing the incumbent, as variations on the saying go.
There are two key points regarding performance that I'd like to mention, and that's that Dolt uses higher-than-expected disk, and we also use more memory than MySQL by comparison. The disk usage is due to our optimization of speed at the sacrifice of disk accretion via temporary storage, which we've decided is a fair trade off considering disk is very cheap (and we're working on making this tunable so users can decide on speed vs disk efficiency). Memory usage is a bit more complex, and I'm not the employee to comment too much about it, but both of these issues are being worked on to reduce their impact. With that in mind, our performance is comparable to MySQL as long as the machine limits are not being reached, however I'd expect us to be a bit slower once those limits are reached, simply due to the extra complexity that we're managing.
Lastly, trust is something that can only be built over time. In 10 years, I'm sure that there will be no doubt of our stability, and at that time I can see Dolt and DoltgreSQL becoming the de-facto databases used for relational storage. Of course, I may be a bit biased :)
Trust is a
Either way, with DoltgreSQL now in development, we will now have solutions for both sides! (Other popular databases with their own syntax, such as Oracle and Microsoft, are not planned for the future at this time)
That's incredibly self-fulfilling. Counter status quo signals self-select out before even talking with status quo vendors.
It's as if driving by rear view mirror based on noticing cars around you, rather than having a traffic helicopter high overhead looking across all the roads and ahead.
Inability to see difference in trending versus point in time is how "incumbents" end up having their lunch eaten. Looking at point in time instead of trend, the challenger isn't a threat until they've crossed over, and by then, well, you've lost 5 years. Of course only looking ahead, you might never get anywhere most people want to go today, and you have to get paid. So the trick is always considering both!
All that said, strong pivot now and we'll be checking your take out.
by sheer numbers, MySQL is probably still dominant but if projects are starting today it's likely postgresql
[1] https://www.statista.com/statistics/809750/worldwide-popular...
[2] https://6sense.com/tech/relational-databases/mysql-market-sh...
[3] https://www.datanyze.com/market-share/databases--272
I'm pretty sure that's the Wordpress effect. How many Wordpress users would buy a database engine that is 99% compatible with MySQL, because it supports branching?
https://github.com/dolthub/dolt
In fact, we've got a blog post about Dolt and Wordpress.
https://www.dolthub.com/blog/2023-08-04-wordpress-on-dolt/
Sure, but my question was about the feasibility of selling support to Wordpress users vs PostgreSQL users. The latter usually care more about advanced features such as the ones Dolt offers.
Don't get me wrong; I think Dolt is a great product. I was simply replying to the 2023 stats posted by k1ns.
https://www.dolthub.com/blog/2022-08-04-database-versioning/
A lot of products have come out that attempt to tackle schema versioning, but none have tackled data versioning before Dolt (https://github.com/dolthub/dolt). In addition, our database isn't forked, it's a full, bespoke solution that can operate as a drop-in replacement for MySQL (Dolt) or PostgreSQL (DoltgreSQL). It's honestly quite exciting technology, so definitely feel free to ask any more questions if you're curious to learn more!
Here is a link to a few use cases as well: https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/
I think Dolt is closer to Neon/Xata. But still there are differences.
IIUC, Dolt is bringing the database feature to Git, while Neon/Xata is bringing the Git feature to database.
Speaking of Bytebase, if Dolt is really good at versioning schema migration, Bytebase value proposition will be a bit less attractive, but not much. It's similar to the Git story, regardless how powerful Git is, people still need GitLab/GitHub for the developer workflow on top of the mere versioning.
Dolt (and eventually DoltgreSQL) handles everything. Branching, merging, diffing, cherry-picking, commits, and more. Working with both the schema and data. On top of that, we also have DoltHub (https://www.dolthub.com/) that's analogous to GitHub, and DoltLab (https://www.dolthub.com/#doltlab) that's analogous to GitLab. We are targeting the entire ecosystem from the bottom up.
https://docs.dolthub.com/introduction/use-cases/data-and-mod...
https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/
That said, we have 99.99% correctness according to sqllogictest:
https://dolthub.awsdev.ld-corp.com/blog/2023-10-11-four-9s-c...
It has data versioning too, using a concept called ‘alternatives’. Do you know where DoltgreSQL and VMDS differ/overlap, functionally?
See https://en.m.wikipedia.org/wiki/VMDS
So it looks like VMDS is a rather unique take on versioning data, where data is viewed as objects. Dolt and DoltgreSQL use the table and row paradigms that the majority of modern relational databases use. Dolt is designed to be a drop-in replacement for MySQL, and DoltgreSQL for PostgreSQL, so that already determines the interface. Regarding VMDS' versioning functionality, I see that is supports merging and conflict resolution at the transaction layer, but I'm not seeing anything similar to the concept of branches, which is something that Dolt and DoltgreSQL supports.
Overall, for modern use cases, it doesn't look like they overlap much. VMDS seems focused more on the spatial case while representing data as objects, while Dolt and DoltgreSQL are traditional relational databases that support versioning all aspects of a relational database.
https://docs.dolthub.com/concepts/dolt/rdbms/replication
Current migration implementations live on the software side and are essentially a folder full of sequential, date stamped, SQL commands to execute. These files are checked into git so they are versioned.
No offense to you or your team or this project, but I obviously trust the Postgres core team far more than you. However, everything is about trade-offs. So, what (briefly) makes this a risk worth taking?
I HAVE needed an old version of a database to check/compare something, but nightly backups have always sufficed, and for disaster recovery I typically have point-in-time recovery set up. So if a migration goes boom I can just say, "ummmm lets go back to a couple minutes ago" which is a nice safety net.
Recovery is a valid use-case for version control, but fully applying it to data opens up possibilities that haven't really been explored before.
I'm curious if you have any plans for local first development. I love the idea of a version controlled database, but I want to use it for local first and distributed data syncing for users who are frequently without internet. It's not clear how you would do this without running the database locally. Do you have thoughts on this? DoltLite?
https://docs.dolthub.com/sql-reference/version-control/dolt-...
as well as the `dolt_remotes` system table:
https://docs.dolthub.com/sql-reference/version-control/dolt-...
We have a ton of remote options:
https://docs.dolthub.com/sql-reference/version-control/remot...
Right now, DoltHub (https://www.dolthub.com/) will still work as a remote but all the SQL there will be the MySQL version. Over time the Doltgres storage format may become more bespoke and we'll have to figure out how DoltHub will work.
But Dolt is 1.7x slower than MySQL, as is written in that Readme.
I'm looking for ways to have effortless and traceable changes to an entry or entries.
I've experimented with ArangoDB but it wasn't for me. Dolt is now the only other VCS for databases I'm aware of.
Are there any alternatives? So I could try more and decide which to settle with.
1.7x MySQL is still very fast. You're talking about .5ms for a point lookup instead of .3ms, it's not something users will notice at typical scales. And we're still iterating on performance, the gap will continue to close over time.
I recently stumbled upon the concept of "bitemporal modeling" (ie. rewinding data "as of") - thought it described well this use case.
There are products that can do something similar for schemas, but DoltgreSQL (and Dolt, the production-ready DB based around MySQL rather than PostgreSQL) applies this to both schemas and data. This is unique to DoltgreSQL and Dolt, and what sets us apart from all other products. One example I like to use that I think really captures this power, is the ability to query data from any commit. We use Git's model of version control, so your database has a commit history and a working set. The working set can be viewed as the "current" commit. When you run queries, you implicitly select the working set, however we expose the ability to explicitly set which commit you're targeting. This means you can do things like run a join query across old and current data!
These situations, and more, and not possible via migration tools, backup tools, or any other kind of tools. This is unique to DoltgreSQL and Dolt, by fully embracing data versioning. This is but one example, and there are many, many more.