HN Explorer

Ask HN: Why are there no easy-to-use highly-available open source databases?

21 points by database_guy ↗ HN
I'm a single engineer who wants to run a database by myself, no cloud vendors involved. I don't really care about scalability to a point, I don't really care about the API to interact with the database. I just want something that I can set up on multiple machines with minimal effort and have the loss of one of those nodes not cause catastrophic failures. I want the experience of something like etcd without being limited to a few gigabytes of data in the cluster (a Terabyte or maybe a couple hundred Gigabytes as a limit would probably be fine). I've spent a lot of time looking through the various options and wrote up my thoughts on why each of them wasn't a good fit.

Postgres: A great database but HA options either have far too many moving parts, require manual intervention, are proprietary, or are Fake Open Source. Seems like there's lack of interest in addressing this because companies that sponsor development generate a lot of value off complex operations

MySQL: This database has some odd behavior but that would be fine if it weren't Fake Open Source

MariaDB: HA Options are poor or abandoned

SQL Server: Proprietary

Oracle: haha

CockroachDB: Fake Open Source

YugabyteDB: Fake Open Source. Special shout out here for not even linking to instructions for how to build the database in the readme.

MongoDB: Proprietary, and even before the license change it was Fake Open Source

Cassandra: Not fun to run, and even though I said I didn't care about API I don't necessarily love how it works. But it comes probably the closest?

ScyllaDB: Fake Open Source

TiDB: Fake Open Source

Singlestore: Fake Open Source

FoundationDB: This one comes close but its beginning as a proprietary database really hurt its community, which is way smaller now than it should be. Could grow into something great if more folks got behind it.

etcd: not suitable for use above a couple GB of data

What do I mean by Fake Open Source? A project that has a large percentage of its contributors beholden to a single organization/entity to me is not really open source in spirit. I'm looking for a project where I can feel confident my contributions won't effectively end up behind some proprietary license down the line if/when the VC backed organization that primarily sponsors development decides it needs to protect itself from AWS. If there's an "Enterprise" product and the organization calls the source code for the main project the "Community Edition" or something like it, it's not Real Open Source. If a single organization shuts down and contributions fall off a cliff (https://github.com/rethinkdb/rethinkdb/graphs/contributors) it's not Real Open Source. There are lots of Real Open Source projects with great communities of users/contributors, but many of the newer databases don't have legitimate open source development communities behind them, in my opinion.

I've probably missed some examples. Mostly, I wonder why there hasn't been a general purpose open source database that does the operations stuff as well as the proprietary databases do. Am I missing something?

41 comments

[ 2.7 ms ] story [ 94.0 ms ] thread
PaulHoule
I'd say Oracle has been a better steward of MySQL than Sun was back when Sun was a separate company.
jka
Yep, your assessment seems broadly accurate to me. I was going to suggest Cassandra until I saw it in the list. I'm {interested in/optimistic about} FoundationDB too, although haven't had a chance to use it in practice yet.

Out of curiosity: what would your preferred choice(s) be to fit these requirements using existing proprietary products?

database_guy
From my research I think Cockroach, Yugabyte, and Scylla are all interesting and solid products depending on the exact API/guarantees you're looking for. For products I don't even have the option to self-host I think DynamoDB and Cloud Spanner are great products, probably among others.
jka
Cool, thanks - yep, DynamoDB seems fairly reliable, and that's also good to hear another vote in support of Cloud Spanner.

Glad to have learned about rqlite from this thread as well.

endisneigh
I’m not understanding why it matters if it’s fake open source or not. Even with “real open source” there’s no guarantee your contributions will be included. Or that the license won’t change in the future.
ummonk
> What do I mean by Fake Open Source? A project that has a large percentage of its contributors beholden to a single organization/entity to me is not really open source in spirit.

Then you should use a different term - something like "community project". The single organization projects are still open source, both technically and in spirit.

> I'm looking for a project where I can feel confident my contributions won't effectively end up behind some proprietary license down the line if/when the VC backed organization that primarily sponsors development decides it needs to protect itself from AWS.

If you're talking about ElasticSearch, I'll point out that Amazon forked it and OpenSearch is not behind a proprietary license, so all contributions made to ElasticSearch continue to be available as open source, with improvements being made.

database_guy
I have contributed code to projects "owned" by one of these types of organizations and after 8 months of waiting for my (very small) patches to be reviewed because I wasn't internal, it just didn't feel like healthy Open Source. Calling them Fake Community Projects as a term works just as well for me.

OpenSearch could be a contender (though I did not have fun running Elasticsearch back in the day). I will keep watching it and see how development goes, it's a good call-out.

mountainriver
I totally feel this but it’s very dependent on the company
elmerfud
I'm not sure if you're just picky or discerning but it seems you can find a reason to exclude anything if all you do is look for reasons to exclude.

Why not just use SQLite with streaming replication? It should fit your bill.

Databases rarely have what you would define as real open source with real contributors because the nature of a database means you need one owner and that owner has to be picky and exclude things. Allowing the wrong commit into MariaDB could introduce regressions that no one even could imagine because the complexity of things.

Even when a database starts out the way you describe with good intentions in order for it to become a widely adopted product it has to be pulled in under a single umbrella to direct it and build it toward its vision. This puts it firmly in your fake open source camp.

database_guy
I am very interested in any project that implements multi-active SQLite clusters with no cluster size limit, do you have any links for me?
azornathogron
rqlite is one of the ones that comes up in discussions. I can't offer a recommendation for or against though, as I've never needed to use it.

Not sure why "no cluster size limit" is important given the goals you listed in your comment btw. The rqlite faq mentions a practical cluster size limit of 9 read-write nodes (higher for read-only replicas).

database_guy
Rqlite is an awesome project but it's not intended for use above a couple GB, like etcd. Sorry, dataset size is what I meant by cluster-size which was a confusing way to put it. You're right, I don't care about the number of nodes in the cluster.
azornathogron
Ah, you're right, sorry. I naively assumed rqlite would cope with similar dataset sizes to sqlite itself, which I think is likely to be limited by how much disk you can put on a single node. But it seems rqlite is much more limited on that dimension, so it doesn't fit your criteria.
otoolep
Full details on what the limits in rqlite may be, and why, are here: https://github.com/rqlite/rqlite/blob/master/DOC/PERFORMANCE...

That said, very large datasets (> 2GB) will work, if you've got the disks and memory to handle it. The link above explains how to do it.

database_guy
Perhaps I'll have to test it out at larger data sizes. The performance document makes it sound like this is here-there-be-dragons territory. I'd love to see how the database holds up at 100GB. Rqlite is a great project and I'm glad it exists!
otoolep
rqlite author here. Happy to answer any questions.

Noting the title of this topic, rqlite is specifically designed to be easy to use. I know of no other HA database that is simpler to operate. Installation is trivial, and clustering is easy. You interact with it via a HTTP API.

And you get all the power and quality of SQLite as the database engine.

otoolep
You can go larger than 9, quite a bit larger. I just suggested 9 as a useful size. But the 7.x series introduced much better support for Kubernetes, so it's much easier now to manage larger clusters.

But larger clusters mean a larger quorum, so larger isn't really better. It depends on what you want. If you want a distributed database where more nodes means more write performance, then rqlite may not be a good fit. rqlite is distributed for fault-tolerance and HA, not for performance.

elmerfud
Just searching for SQLite replication will get you several options to research to fit your requirements. The concept of multi active and no size limit you would simply solve with a sharding mechanism that you layer on top through a data access method.

But if you're looking for some monolith project to provide all of this for you you're going to be in that territory of that fake open source. Because again to build all of this into one tool that has everything from the data access layer all the way down to the metal underneath to ensure that it's replicated and highly available is not a trivial task. Especially if you want things like acid compliance arbitrary joins between data structures and so forth.

The file system is effectively a database of things with an API that accesses those things. Simply a file system of properly named files and directories could meet your needs depending on what those are.

It's the complexity of your access model and methods that will determine what fits your bill.

database_guy
Yeah, sounds like most folks agree this doesn't exist simply because it's very challenging to implement.
iAm25626
PostgresSQL - Not sure which HA solution you had experience with: https://patroni.readthedocs.io/en/latest isn't too bad. old adage comes to mind. Fast/Cheap/Good - pick 2 HA design are not all created equal. rubber stamp something HA often give false sense of security. HA to me is explicitly defined risk(down time) tolerance. For each of 9 it get more complex and cost goes up. Most commercial DB with HA are opinionated which is often the opposite ethos of open source
znpy
The reason: ha for databases is hard.

However galera cluster and/or percona xtradb cluster work remarkably well, considering they’re open source.

toast0
Also, HA is all about tradeoffs, and there's lots of reasonable selections, so there's no way to make that easy.
ddorian43
> YugabyteDB: Fake Open Source. Special shout out here for not even linking to instructions for how to build the database in the readme.

All the features are open source. Here is how to build from source https://docs.yugabyte.com/latest/contribute/core-database/bu....

> What do I mean by Fake Open Source? A project that has a large percentage of its contributors beholden to a single organization/entity to me is not really open source in spirit.

Well, somebody gotta start the project, no? Feel free to contribute though. Since it reuses PostgreSQL, it directly inherits the "postgresql community commits". The same with being a fork of Apache Kudu fork.

> If there's an "Enterprise" product and the organization calls the source code for the main project the "Community Edition" or something like it, it's not Real Open Source.

The "Enterprise" edition is "just" some scripts that make deployment & monitoring easier (and includes 24/7 developer support). All c++ features are open source.

And it's still young. You can't compare against PostgreSQL that has 20+ years of being available.

database_guy
Fair point re: "someone has to start the project". I think Yugabyte is neat, and I want to invest in it as a platform, but I can't help but look at the project and feel like it's not set up for success should the Yugabyte the company fail. I hope I'm wrong and in 10 years it looks something like the Postgres community.

I also did eventually find the docs for building the database. It just feels odd to have a roadmap in the readme but no actual information for getting set up on development. It's one of those things (and I grant it's a very small thing) that makes me feel like the platform isn't really for the community.

ddorian43
> should the Yugabyte the company fail

Probably true as of now. But they have enough funds for some time to seed the project in the worst case scenario.

> It just feels odd to have a roadmap in the readme but no actual information for getting set up on development

Because 99% of developers just want to use it instead of contribute in c++. And the docs are pretty big. We'll include a link for contributors/building docs in the readme.

It takes like 10 years to create a new database ready for production. And not everyone can contribute a quick patch to it, it needs some extensive skills.

But we're extremely open to outside contributors. And we've accepted fixes/features outside of the roadmap. And the license is free enough that you can build a competitor with it.

Source: I work there.

[deleted]
(comment deleted)
neoCrimeLabs
You are missing something.

You list several completely different types of databases and then mention you are looking for a "general purpose database".

This tells me you do not understand the problem you are trying to solve, therefore do not know how to define the requirements nor features you require to solve it.

It also tells me you do not understand the difference between different types of databases, nor the niche they fill, let alone why their HA functions very differently.

database_guy
My point is that for small scale for my uses the differences between these databases are mostly academic. I am fully confident in my ability to model data around their various constraints and get them to do pretty much anything. I also understand why their HA functions differently. To be clear, there are projects that, governance/development community aside, hit all the right spots technology-wise for what what I'm looking for. These projects just seem to have far less healthy communities to me than others and I'm curious why that seems to be the case.
otoolep
rqlite https://github.com/rqlite/rqlite

I'm the creator of this project. While it's not going to work super well at very large datasets, it's explicitly designed to be trivial to deploy, and very easy to operate. You can get it up and running in seconds, and clustering seconds later. My practical experience with databases told me that operating the database is at least as important as performing queries with it. So I put a lot of work into easy clustering, clear diagnostics, and solid code.

d_t_w
I have deployed and run Cassandra myself, basically as you describe.

The first 'dev' Cassandra install was three nodes. I downloaded the .tar.gz, installed it, started each process in turn on each node with the required configuration.

That was 2012 and there is a chance that cluster is still running. It was low-volume in terms of data. TTL configured so it would never run out of disk. Never had any issues in particular. I used it for ~5 years before concluding work with that client.

The problem in that case was Cassandra proliferated in that small company, they didn't build any particular expertise beyond me, and in the end I was being pulled into discussions from different teams, different products, split across about 13 clusters. Wasn't much fun - but that wasn't the DB's fault.

database_guy
Cassandra does feel the best here to me. Thanks for the experience report, maybe I should test out how it runs when confined to somewhat small scale.
techdragon
The community is still slowly recovering from the collapse of the company but RethinkDB is worth considering in your analysis.
database_guy
I really liked RethinkDB (maybe even loved?), but its demise is one of those cautionary tales that makes me wary of these single-corporate-sponsor databases.
techdragon
Absolutely agree, it’s why I’m so hesitant to let myself like CockroachDB, I’m not a raw SQL person, database choices are a huge investment for me. PostgreSQL and SQLite are the only databases I can recommend without hesitation.

I’ll admit I definitely fell all the way into “loving” RethinkDB. I still have some sticker bundles from a meetup group merch package. It’s been an agonisingly slow recovery from the company’s demise, but I still hold onto some hope that it can recover and become a multi-stakeholder database with a steady development like PostgreSQL for the horizontal scale document database world.

newaccount74
It's not clear to me why you need a distributed database in the first place. If it's just for general purpose small scale projects, does it really matter if your database is down once a year?

I run a couple of Postgres databases on cheap linux VMs, for various projects, and they have been running smoothly for years. The only problem I had was two times when the disk was full. If I had multiple nodes they would all have been full...

Github has been down more often than my Postgres databases.

HA adds so much complexity and tradeoffs that I would really think hard about wether it's worth it for your use case.

asadawadia
What do you want specifically in a DB? Consistency? Sql or nosql? HA or sharded? homogeneous nodes?

Jbtw, the risk of nodes dying is quite exaggerated. DO/Vultr/Linode all provide 99.99% up time

snikolaev
Manticore Search is not in the list:

* 100% open source (GPLv2)

* Easy replication:

  - node 1: CREATE CLUSTER c; ALTER CLUSTER c ADD tbl; 

  - node 2: JOIN CLUSTER c AT 'host:port'
* Easy HA: CREATE TABLE dist TYPE='distributed' agent='...' agent='...'

* Real alternative to Elasticsearch in terms of built-in full-text search capabilities, but easier to use.

* Works fine with small and large data volumes: - with in-memory storage for smaller data - with columnar storage (separate library, Apache2 license) for big data that doesn't fit into RAM

* Does analytical queries well

* Not fully ACID-compliant (as well as Elasticsearch, Clickhouse others)