This is a real trend, at least anecdotally in organizations I work with, and it is being driven in large part by the extraordinary cost -- both time and money -- of moving and reorganizing data within and across systems, particularly as the volumes and velocity inexorably increase. Unnecessary data motion is an operational killer.
Working against this trend is that you can't make a database competently "general purpose" that was not explicitly designed to be this way in the first place. There is a lot of friction created by the impedance mismatch between a database kernel and the extensions it was never designed to support. Also, even for relatively general-purpose databases like a modern RDBMS, "general purpose" encompasses a much wider array of features today than were anticipated by anyone when these databases were designed.
I am skeptical that any current database can meet that criteria through evolution. Existing relatively general databases are insufficiently scalable for many data models and the most scalable databases have internals that are heavily overfitted for a single type of data model to the exclusion of ever having good performance for most others. Most of which can be explained by none of them having been designed with the end state we are talking about here in mind.
It should be added that building a database specifically for scalable generality would be extremely expensive. Reductions in generality are a major mechanism for reducing engineering cost in database engines. I'm not sure you could even find the resources to undertake such an endeavor today. Definitely not the kind of thing where a couple people can build an MVP working nights and weekends.
We have general purpose databases. They're called filesystems. You can literally store anything and everything you like, and hardware is ever-shrinking in cost and performance.
The problem you're dedscribing is political: How to get everyone across cultural and language barriers agree upon what the bits mean?
There is no technical obstacle to everyone accepting a universal API. In many ways, we're on our way there already. The Internet is full of standards and protocols. Yet it shunts knowledge around the world caring only of a few numbers.
It might be something fundamental, like that each aspect of reality has to be attended to by a sentience. Like every person is responsible for maintaining some part of the universal standard. Maybe general-purpose-anything is flawed as a concept, and we must come to understand that what we seek cannot be automated in a meaningful way.
Maybe in the age of Turing et al there was so little knowledge it all seemed computable. Now we're drowning in it.
This is my experience as well. At the end of the day, most line of business applications can just use SQLite and be done with it. One project, one executable, one database, one model, one machine. Very rarely does the business actually need the kinds of fantastical things that are routinely bandied about on hackernews. Now, that isnt to say these things arent fuckin awesome and that we should stop talking about them, we just need to be realistic about the scope and applicability of our tools.
In terms of a universal API, we are kind of doing this in our shop. We are building 1 unified domain model, which is approximately in 3rd normal form. All of our software is required to use this model to represent the facts/state of whatever is happening. Changes to this model go through a semi-formal review process, just like you would expect with an old school SQL setup in a big shop. Our model can be mapped one-way to SQL for purposes of online business rule evaluation or offline reporting.
The benefits of having 1 standardized model are really cool after its been in place for a while. You will notice your non-technical project managers developing an awareness for which top-level types may be implicated in certain matters. Having one slow-changing way to represent the business at hand makes it really easy for people to start picking things up. When its scattered to 50 repositories and 10 different languages, this is a bit trickier.
It's kind of the same argument for/against a universal programming language. The sheer creativity of all the peoples working in the same language, eclipsing the few lost concepts we are unable to represent with our current model.
If we want a robust model, we have to include uncertainty within it. As long as computers fail by being sure about the error, a human is always needed to fix the problem.
It would have to be an almost philosophically mathematical standard. How we do that when half the world measures physics by completely different metrics than the other one?
At least we have Unicode. God bless Unicode. Solving the problem from a computer-linguistic viewpoint: Let's get the language into the computer and we can worry about what it means later!
> We have general purpose databases. They're called filesystems. You can literally store anything and everything you like, and hardware is ever-shrinking in cost and performance.
And they're even more limited than relational databases when it comes to the ability to scale out, or have good consistency semantics without deadlocks. Just implementing a basic mutex on a (general) filesystem is nigh-on impossible. And while AFS is impressive, there still aren't really any true master-master filesystems, much less filesystems that can autoscale.
Consider the hubris behind a term like autoscale. You're talking about a machine that makes more machines, adapting to its environment, and all that without human intervention.
I think it's far more likely we'll just discover that, indeed, no general solution exists, and that humans must take personal responsibility for tending to their digital gardens. We certainly have a lot of spam to weed out.
That's nice but autoscaling mongodb clusters have been working for what, 5+ years at this point?
There are limits to what you can do safely with autoscaling, sure. But a datastore without that capability will never be a universal datastore that can cover all use cases.
This whole 'general purpose' term is meaningless. If you're talking about NoSQL vs SQL (ACID), you're just looking at trade-offs. Data integrity and distributed systems are always in contention. Of course, the moment you don't rely on types and structure constraints, you can have more performance and easier replication. You can't just solve this problem through evolution, until we have quantum computers, but all living projects are evolving towards some overlapping features and happy mediums.
The thing that seems to be happening over time is that those who optimize for performance, want integrity, and those who start with transactional guarantees want performance. So I guess, there could be some hype train pendulum swinging.
> Reductions in generality are a major mechanism for reducing engineering cost in database engines
Yeah, just look at snowflake for a great example of this. They outsource the entire bottom half of the database (making data durable) to a blob store (S3, GCS, etc.). That is a huge simplification for them. They don't need to write their own durability and high availability components (no write ahead log, no recovery or replication protocols, etc.). It also makes it completely impractical to run lower latency small write transactions vs snowflake because the data needs to hit blob storage before the transaction can commit (and blob store writes are 10 to 100s of ms of high variability latency). Being a more specific case database (a data warehouse) let them make this trade-off. More general purpose systems can't make this type of trade-off. I wrote a bit on this topic a few months ago if anyone is interested [1]
14 comments
[ 5.9 ms ] story [ 45.6 ms ] threadWorking against this trend is that you can't make a database competently "general purpose" that was not explicitly designed to be this way in the first place. There is a lot of friction created by the impedance mismatch between a database kernel and the extensions it was never designed to support. Also, even for relatively general-purpose databases like a modern RDBMS, "general purpose" encompasses a much wider array of features today than were anticipated by anyone when these databases were designed.
I am skeptical that any current database can meet that criteria through evolution. Existing relatively general databases are insufficiently scalable for many data models and the most scalable databases have internals that are heavily overfitted for a single type of data model to the exclusion of ever having good performance for most others. Most of which can be explained by none of them having been designed with the end state we are talking about here in mind.
It should be added that building a database specifically for scalable generality would be extremely expensive. Reductions in generality are a major mechanism for reducing engineering cost in database engines. I'm not sure you could even find the resources to undertake such an endeavor today. Definitely not the kind of thing where a couple people can build an MVP working nights and weekends.
The problem you're dedscribing is political: How to get everyone across cultural and language barriers agree upon what the bits mean?
There is no technical obstacle to everyone accepting a universal API. In many ways, we're on our way there already. The Internet is full of standards and protocols. Yet it shunts knowledge around the world caring only of a few numbers.
It might be something fundamental, like that each aspect of reality has to be attended to by a sentience. Like every person is responsible for maintaining some part of the universal standard. Maybe general-purpose-anything is flawed as a concept, and we must come to understand that what we seek cannot be automated in a meaningful way.
Maybe in the age of Turing et al there was so little knowledge it all seemed computable. Now we're drowning in it.
This is my experience as well. At the end of the day, most line of business applications can just use SQLite and be done with it. One project, one executable, one database, one model, one machine. Very rarely does the business actually need the kinds of fantastical things that are routinely bandied about on hackernews. Now, that isnt to say these things arent fuckin awesome and that we should stop talking about them, we just need to be realistic about the scope and applicability of our tools.
In terms of a universal API, we are kind of doing this in our shop. We are building 1 unified domain model, which is approximately in 3rd normal form. All of our software is required to use this model to represent the facts/state of whatever is happening. Changes to this model go through a semi-formal review process, just like you would expect with an old school SQL setup in a big shop. Our model can be mapped one-way to SQL for purposes of online business rule evaluation or offline reporting.
The benefits of having 1 standardized model are really cool after its been in place for a while. You will notice your non-technical project managers developing an awareness for which top-level types may be implicated in certain matters. Having one slow-changing way to represent the business at hand makes it really easy for people to start picking things up. When its scattered to 50 repositories and 10 different languages, this is a bit trickier.
If we want a robust model, we have to include uncertainty within it. As long as computers fail by being sure about the error, a human is always needed to fix the problem.
It would have to be an almost philosophically mathematical standard. How we do that when half the world measures physics by completely different metrics than the other one?
At least we have Unicode. God bless Unicode. Solving the problem from a computer-linguistic viewpoint: Let's get the language into the computer and we can worry about what it means later!
And they're even more limited than relational databases when it comes to the ability to scale out, or have good consistency semantics without deadlocks. Just implementing a basic mutex on a (general) filesystem is nigh-on impossible. And while AFS is impressive, there still aren't really any true master-master filesystems, much less filesystems that can autoscale.
I think it's far more likely we'll just discover that, indeed, no general solution exists, and that humans must take personal responsibility for tending to their digital gardens. We certainly have a lot of spam to weed out.
There are limits to what you can do safely with autoscaling, sure. But a datastore without that capability will never be a universal datastore that can cover all use cases.
The thing that seems to be happening over time is that those who optimize for performance, want integrity, and those who start with transactional guarantees want performance. So I guess, there could be some hype train pendulum swinging.
Yeah, just look at snowflake for a great example of this. They outsource the entire bottom half of the database (making data durable) to a blob store (S3, GCS, etc.). That is a huge simplification for them. They don't need to write their own durability and high availability components (no write ahead log, no recovery or replication protocols, etc.). It also makes it completely impractical to run lower latency small write transactions vs snowflake because the data needs to hit blob storage before the transaction can commit (and blob store writes are 10 to 100s of ms of high variability latency). Being a more specific case database (a data warehouse) let them make this trade-off. More general purpose systems can't make this type of trade-off. I wrote a bit on this topic a few months ago if anyone is interested [1]
[1] https://www.singlestore.com/blog/separating-storage-and-comp...