In theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of things and their relations.
Practically speaking, the scope of a given database/schema is generally restricted to one business or problem area, but even this doesn't matter as long as the types aren't aliasing inappropriately. You could put a web retailer and an insurance company in the same schema and it would totally work if you are careful with naming things.
Putting everything into exactly one database is a superpower. The #1 reason I push for this is to avoid the need to conduct distributed transactions across multiple datastores. If all business happens in one transactional system, your semantics are dramatically simplified.
> Putting everything into exactly one database is a superpower.
Especially this.
$1M big iron DB server is much cheaper than a redundant array of inexpensive databases when people come to actually use the data; be it developers, analysts or leadership, everybody saves time, except perhaps a few DBAs.
I've worked at two of them -- all data, sans pointers to content addressed immutable object storage, is stored in a single database. It worked well for us; it's not rainbows and unicorns wow all of our problems disappeared but we got what we needed with sufficient performance for an app that was not well optimized.
There were challenges like the difficulty of working with a few huge tables that existed since the company's inception and were practically infeasible to do any schema changes on, but that's more a modeling issue. It was also a single point of failure service. If the db goes down so to does everything but constant sense of impending doom it was less of a problem than I thought with ndb.
Sorry, I mean a single point of failure service. I guess maybe the term would be large blast radius. The database implementation was HA to almost paranoid degrees but if we lost the database it was a total outage every service down for all customers. If the DB had issues there was really no sense of a partial outage.
It is good to see people on HN are supporting this single db approach. And with development such OrioleDB, it would solve the horizontal scaling. And all the access to data are native.
Amazon's entire system ran on a single Oracle cluster up until about 2008 (they were still at the tail end of phasing that out when I was there in 2012).
In my time at Samsung, we were tracking most of the authoritative state for the factory in a single monster Oracle DB instance. This made troubleshooting systems integration problems fairly trivial once you had some business keys to look for.
There are so many crazy tricks out there. You would be amazed at how much abuse an old school RDBMS engine can take on modern hardware. In-memory OLTP on servers that can host terabytes of RAM completely changes the game. Disaggregated compute & storage provides seamless OLAP capabilities on the same SQL connection instance for the most modern providers.
Oracle (the software) isn't a database, though. It is a database management system. It seems highly unlikely that it kept all the data in one place. Different workloads generally need different database structures. And Amazon circa 2008 was complex enough that it no doubt had a gamut of different workload needs, even if that was all managed under one management system.
Really, as it pertains to this, it is a question of "where do I want to move the complexity to?", not "can I eliminate it?"
As someone bootstrapping a business, my cost structure favors horizontally scaling databases with low/no fixed overhead vs vertically scaling databases with high overhead. And I don’t think this is exactly an uncommon situation - plenty of people, even within large well-funded organizations, would have a hard time justifying such a large expenditure if it were uncertain that it’d be fully utilized.
It’s also a lot easier for me to handle tenancy/isolation/different data “realms” this way. And I sleep better at night knowing a mistake (I have no dbas, and I don’t have time to invest in tons of safeguards as a solo dev when there are much higher EV tasks) won’t nuke absolutely everything.
For small companies, I've never started with horizontal scaling.
With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries.
If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.
I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling.
It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
You also often have to fight for the right thing, then any problems that do happen are your fault.
When the dumbshit over engineered "cloud architecture" that was pushed on you (managers don't get promoted or switch companies for a big raise by reducing their headcount... no, they get promoted by building a big team for a successful [please don't check in on how it was doing two years later...] "cloud transformation") has all kinds of problems and the costs shoot into the stratosphere, while five servers on a rack would have granted greater uptime and performance in practice for your usage patterns, and lower costs... it's not your fault.
Have some bad luck and that simple solution you pushed for happens to experience an unlikely failure causing a significant outage in year 1, and your head's on the chopping block.
What I experienced is that it doesn't matter if I pushed for the right thing or inherited some overcomplicated shit. In case of any problems it's always my fault anyway. You just can't win.
If you want the latitude to affect meaningful change, then you are also going to have to accept all ownership surrounding that thing and likely far beyond. I know it gets some flak (for good reason), but concepts like extreme ownership are fundamentally how I am able to tolerate this "everything is my fault" experience.
If everything will be my fault in the end, then screw it - I will go ahead and take ownership/control of all of those things and make sure they don't come back around to bite me in the ass. If someone questions my authority, I always cast it in terms of "how many weekends with your family are you willing to sacrifice to support that idea?" Oftentimes this is all it takes to get those idle suggestions put in the bin so we can move forward.
Solving database circus in a real business environment requires disengaging all of the safeties and getting very messy with your free time, stress levels and other humans. It's a front-loaded suffering though. If you actually know what you are doing and get through the schema refactor (and it sticks), everything is down-hill from that point. Getting one schema to rule them all and then having the team learn that schema == new common language by which you can communicate that did not exist prior. In our business, non-developers are just as aware of our tables, columns and relations as developers & DBAs are. In some cases, they have more detail about how a field is used than I do.
I wish it would work in practice like that. Because in reality you can't just start making unapproved changes unless you really want to get fired quickly. But you can't get approval either. But you are 'responsible' for keeping it up and running, too.
This industry is way too much of a fashion show. And too many devs are consumers of new technology in a similar way.
After 20 years I am actually getting a bit skeptical behind the "you should constantly be learning" mantra. If you are constantly learning new, tech, you will never be an expert in anything. (That statement is very much a generalization and I am sure there are tons of people that will point out issues with it).
To elaborate, I’m using managed databases that abstract away scaling so that my UX as a developer is that they are scaling horizontally (in data, and cost).
While I’m pre-launch and doing testing, using a more traditional vertical model is money out of my pocket. And it’s not something I want to maintain after I launch either.
Also, I’m creating something (a freemium game) where as a solo dev it will only be worth it for me to continue working on the business if it sees explosive growth. If I’m lucky enough for that to materialize, there will be at least a few months before I can hire fulltime help for the product (while also being busy doing tons of other stuff) - it would be terrible to have to handle tons of db operations tasks or re-architect things to support growth during that time.
Basically, vertical scaling is optimizing for the wrong level of usage. It’s actually the same for many of the snarky “your startup doesn’t need horizontal scaling” comments here - if your states are {building, huge growth, failure} then horizontal scaling is perfect because it keeps your costs low while you’re building and handles the huge growth phase more gracefully. Yes, maybe you’ll never get to the huge growth phase, but the entire business is oriented around either getting there or failing.
I certainly didn't mean my comment to be snarky. However, I do want to be realistic. There's an exception for every guideline. Maybe yours is that exception. I worked at a company like you described, where we had a few tens of thousands of users, but were actively in talks with a partner that would have given us international press, and who would have taken us to about 20 million concurrent users a month later. (Imagine a beer company telling their customers to use our app to interact with the Super Bowl broadcast. That sort of thing.) We spent a lot of time talking about horizontal scaling because that was the whole point of the company. Maybe yours is like that, too.
But the vast majority of small companies are not like that. They acquire one customer, then another, and another, and grow N<=50% per year, with tons of headroom on default app configurations. For those companies, worrying about explosive, global scaling instead of, you know, building their business, is time they can't get back.
I worked for a company. The system was basically a crud web app, where users could upload medical imaging data, and send it to be processed in the background.
We had the web app set up on Kubernetes, yet the background jobs had to be manually managed by admins at the company. There were less than 300 users in total and the only time Kubernetes scaled beyond one pod was when something was broken. When I joined, they had just implemented the Kubernetes stuff. I thought it was kind of overkill, but as the new guy I didn't feel it was my place to point it out. Buy hey, some people got some nice experience on their resumes.
I know this well. I've also head to argue for 'even if we become a monopolist is our market, the current service without scaling will keep up (we can literally get a reasonable guess of the total number of potential users), why are you implementing holizontal scaling?'
Do you ever get to a place where you actually have to scale it? Like, the PC from my teenager age would probably be more than fast enough for 80% of companies' data.
Also, are you sure your data is actually in a correct view across all these "realms"?
Basically I’m optimizing for viral growth, otherwise the business probably isn’t worth it. I haven’t launched yet but I estimate that at 100k DAU vertical scaling/using a single instance would become a nightmare because of throughput and latency rather than data size.
I’m admittedly using a strange architecture for my use case and I realize now commenting here opened too big a can of worms as explaining exactly what I’m doing would derail the thread. Suffice it to say, my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data.
But yes, using a more horizontal database strategy makes it very easy to manage data across realms. That’s one of the main benefits. A single DB would be much harder as far as isolation and separating test/production traffic (assuming this is what you mean by views) than having multiple separable dbs that only talk to dev/staging. And I can easily wipe out and isolate dev and staging this way. I’m frankly shocked people would advocate a single db that doesn’t allow you to do this.
That's chump change size even for a medium EC2/RDS instance, which should be capable of tens of millions of queries a day without the CPU or disk complaining at you (unless all your queries are table scans or unindexed).
> my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data
It doesn't matter, it's still just bytes. What will matter is your query pattern relative to the databases query planner efficacy, and how updates/deletes impact this.
> makes it very easy to manage data across realms
You can just as easily do this at first as separate databases/schemas on the same physical server, with different users and permissions to prevent cross-database/schema joins so that when you need to move them to different machines it's an easier process.
Everyone I know that has tested isolated multi-tenancy that wasn't dependent on legal needs ended up abandoning this approach and consolidating into as little hardware as possible. Heap Analytics had a blog post a few years ago about this, but I can't seem to find it.
Regardless, hope you success in your endeavor and that you come back in a few months to prove us all wrong.
If it's a game, transactions are usually either very few per player per day (login, begin or finish playing a level, say something in a chat, spend money on a loot box, etc.) or easily sharded (e.g. 30 commands per second for each of 10 players in a multiplayer level that lasts 10 minutes, not for each player all the time).
> Basically I’m optimizing for viral growth, otherwise the business probably isn’t worth it. I haven’t launched yet but
If you have not launched yet but are optimizing for facebook-scale, that's not the optimal approach.
I can't comment on your database experience since I don't know it, but the vast, vast majority of people underestimate by orders of magnitude what a database can handle.
If you're not a large public company we all know about (and you're not, if you haven't launched yet), you don't need all the horizontal scale you seem to be building.
I remember joining one company (still a startup but a large one about to IPO). My day#1 briefing was about how this one database was in urgent need of replacement with a dozen+ node cassandra cluster because it was about to exceed it's capacity any second now. That was to be my highest priority project.
I took some measurements on usage and capacity and put that project on the backburner. The db was nowhere near capacity on the small machine it was running on. Company grew a lot, did an IPO, grew some more. Years later I left. That db was still handling everything with plenty of headroom left to grow more.
Something operates the database. That thing is going to need a database.
Postgres and etcd are very different. SQLite is very different. Look at this microk8s thread - https://github.com/canonical/microk8s/issues/3227 - as a canonical example. Context is they use dqlite, which is multi-master SQLite with raft. Here people come to actually use the data and it’s a disaster. Just use etcd!
Then people will deploy Postgres on top of Kubernetes. Great. Maybe they will deploy it differently, and their database is a bunch of files in /etc/. Whatever. But you can’t run Kubernetes on Postgres, you wouldn’t want to anyway, the choice to use etcd is very thoughtful. This is an environment designed to simplify the layers of bootstrapping, Kubernetes’s superpower is that it runs as much of its own stuff on top of itself as safe as it is to do so.
It still needs a database that is going to be separate from your applications, and unfortunately, for all interesting applications, the line between deployment and business-logical concerns is blurred.
So IMO, you will have multiple databases, because logical and deployment concerns are intertwined; deployment concerns need databases; and thus to solve chicken and egg, you’ll have at least 2, hopefully the 1st is super easy to administer.
Perhaps you disagree. Maybe you should use dqlite as the only database. It’s so naive to think anything is CP as long as there’s consensus in front of it. And if there’s no strong opinion about the arcane issue I just said, worse if it’s meaningless gobbledygook; or if the strong opinion is “use Postgres,” the data is screwed and the users are screwed.
It's also entirely practical for everything that's not twitter-like, i.e almost everything. But the most noise comes from people with the (admittedly more interesting) use case requirements at the data store level.
So long as your application doesn't throw space and performance away at every opportunity, vertical scaling goes really far these days.
It can be, but I've discovered that in tech it's common to assume that your employees' time costs $0, for some reason. I couldn't even begin to count the number of times I've had to walk a manager back from deciding that we should spend a person-month of engineering time to develop an in-house solution that lets us avoid a $500/year licensing fee.
I see this a lot in RDBMSes. I've used plenty of 'em, and I have nothing against projects like PostgreSQL; they're amazing, especially for the price. But computers cost money, and maintenance time costs money, and development time costs money, and I know from experience that I can squeeze a lot more performance out of the same hardware using MSSQL than I can PostgreSQL, and I'll have an easier time doing it, too. I can even point to some of the specific design and feature differences that lead to the discrepancy.
It's to the point where, whenever I see an article talking about the limitations of relational databases, I automatically mentally insert a "that you can get for free" into the sentence.
Being able to model a use case with tuples and relations does not mean the database can meet the performance requirements of that use case. If it can't meet the performance requirements, then the use case is unsupported. It's the same way how no single data structure or combination of data structures can support all regular programming use cases. Sometimes you need a map, sometimes you need a list, sometimes you need a set, sometimes you need a combination, and sometimes you need something completely different.
Sometimes you just need a DBA to help you optimise your DB for your workload and future growth. In general, developers make shitty DBA's and vice versa.
As a DBA, who has managed enough monolithic databases at mid and large sized organizations, there are enough safeguards in today's date to avoid the scenario you described - backups, read replicas, replication to avoid the need for unnecessary distributed databases.
Every DBA I've worked with has had performance tuning as one of their top skills, both at the installation-level and query-level. Sometimes it's optimizing a query using some hard-earned knowledge of the RDBMS, and sometimes it's keeping logs and data on separate storage.
Yep, you need Google-scale for your shitty startup from the get-go. Otherwise why bother? Especially now, when who have single servers with only a few terabytes of RAM at your disposal.
Many of the problems with databases that I outlined in that post are about how they create complexity, which is not necessarily related to performance or scale. Complexity kills developer productivity, which reduces iteration speed, which can be the difference between an application succeeding or failing.
Interesting, what if one creates multiple databases, but on the same instance (or even process group). Can these conflicting issues resolve somehow? Is there a super-proxy which would only log transaction phases and otherwise offload jobs to database servers (and sqlite wrappers)?
I can imagine Codd saying the exact inverse: any sufficiently complex data model quickly becomes intractable for developers to assemble ideal indexes and algorithms together each time in response to new queries, which kills productivity and reduces iteration speed. Particularly as the scale and relative distributions of the data changes. The whole idea of declarative 4GLs / SQL is that a query engine with cost-based optimization can eliminate an entire class of such work for developers.
Undoubtedly the reality of widely available SQL systems today has not lived up to that original relational promise in the context of modern expectations for large-scale reactive applications - maybe (hopefully) that can change - but in the meantime it's good to see Rama here with a fresh take on what can be achieved with a modern 3GL approach.
In the experience of my department, event sourcing has brought complexity, not taken it away. Theoretically, when done right, by skilled and disciplined teams and to store the state of one system to build projections off of and not to publish state changes to other systems, I think it might work. But in most cases it's overkill. A big investment with an uncertain payoff.
At least, if it can be modelled by some type of finite database stored on disk or in memory, then you can store the entire contents of that database as one binary blob.
Now you just need one tuple ("our_system", "the_db", the_binary_blob)
Practically? I've never had someone throw a problem at me I couldn't model in SQL. Not saying I can guarantee performance, but in terms of abstract modeling I've never encountered something that can't be done "clean".
I'd like to use the analogy of painting. What can't you paint on that canvas that isn't trivially-hindered by the available media? Can you actually describe that space of things?
In my estimation, premature optimization is why most people can't work in these terms. You wind up with a tuple like WorldObjects and immediately go off the deep end about a billion row scale before you are done naming things.
No. Many real domains like natural language are too complex and fuzzy to accurately model with tuples.
To be truly universal, your model needs to be computationally stronger - it needs to be turing complete. For example fractals could never be modeled by any number of tuples, but can be perfectly represented by very short computer programs.
I think there is some "Turing-completeness" for this category as well, that is quite trivial to achieve, though the practical usability may be negligible.
One hacky equivalence I can come up with is that you have a tuple with a single string value, and encode something in that string.
It's a dream, a truly beautiful one, but it never works (does it? has anyone pulled it off?). Taking it to an extreme, the entire world needs one integrated database and user authorizations.
Obviously data structures vary, performance is required, and it becomes a bottleneck because it's so critical that mortals can't touch it and every change must be ultra-safe. And then there's security: what if someone finds a way from the development bug section to the HR or executive-only financials sections? Generally, anyone who has tried to implement universal ERM systems knows how difficult and painful integrated systems can be.
But those are the extremes. I'd be interested to know how far people have gotten, and how they did it, pursuing this ideal? I've never seen a business run on one. How about a personal knowledge-management system? Does everything fit? Do you still use spreadsheets for something 'quick', text files for free-form, etc.?
I once toyed with the idea of viewing all the available compute and storage in the world as one big decentralized computer, with multi-tenancy and auth baked into the OS.
> Putting everything into exactly one database is a superpower. T
Amen. Previous co. had a cargo-cult microservices setup including a separate DB for each app. This made things unnecessarily complicated and expensive for no business benefit (definite resume-padding benefit). Lesson: Don't complicate things until you're forced to.
The thing with the microservice + do separation problem is that by the time you’re forced to do it, the cost to migrate and split data out of a single source into many (potentially all at once) is extremely great. Last at work week a trio of services which rely on the same physical db had a problem where they were both locked out because of a problem with the logic in one of them. This caused an outage that lasted 5 hours and resulted in a (possible) loss of several months worth of their dev team’s funding. At a certain scale you try to proactively lessen the impact of possible events like these because the math works out.
If you’re at that scale, then great, do that. But I also fully agree that cargo-culting that kind of decision out of the gate can be a massive waste of money.
Hard disagree. You don't need different physical databases (though it may become necessary at some point), but you should absolutely value data isolation when you're developing smaller services. Oh you need to rename a column and 6 services call in to read it write this table? You took a 1x solution and made it 5x more difficult to make this change and coordinate, etc. If instead 5 services asked the 1 holder of truth service about the data instead, all of 1 service needs to change when you update the database schema. You can then pick and choose when each dependent service is moved over to the new schema's representation without blocking all services from a simple upgrade path .
If on the other hand, you're fine yolo bringing down your entire tech stack for potentially hours force upgrading all services at once, then you probably shouldn't be using microservices to begin with.
(And yes there are strategies to mitigate needing a big bang for renames as well as long as writers are all consistent, I just wanted to describe why everyone touching the same data stores leads to terrible outcomes like people being unable to make changes because "it's hard", which nobody says out loud but it's definitely visible in the tech stack).
I work in ERP space, which is the ultimate uncool from typical HN perspective; but sometimes "uncool" has a Venn diagram overlap with "mature".
A modern top tier RDBMS is an incredibly powerful, fast, mature, stable tool - when wielded wisely by the knowledgeable wizard. To be fair, the developers must have a solid understanding of programming language, and business domain, and SQL / data dictionary / ERD - which I've always taken for granted but eventually realized is not the norm.
I also work on this space, and kind of see the value of this idea.
Having a log store make sense, and having a super-VIEW replacement that can re-compute the data (so I can denormalize o build secondary tables/indexes that could or not be stored) is a common pattern.
That is fine.
What is NOT fine, and that normal DBS avoid, is to add "eventual consistency".
I NEED to get exact counts, sum, avg, and RIGTH NOW, not a second later.
The more "eventual" the data become the worse things are, and that is the major way to add bugs and problems.
One of the reasons FK are important is they eliminate the chance of getting the relationships wrong, but also, to get it right NOW.
Agreed. Putting data for all your use cases in a single database is a huge deal. Having different databases for each part of the application bloats so much so easily from a cost, complexity and skill set standpoint. Also, makes it extremely hard to debug issues when you have to chart the path of data through 5 different tools.
There is the challenge of workload separation and scaling each component separately but that can be resolved by pulling out challenging workloads into their own "database" albeit on the same stack.
> If all business happens in one transactional system, your semantics are dramatically simplified.
100% agreed. One of the biggest issues SQL databases have faced is that the scope & scale of "one transactional system" has evolved a lot more quickly than any off-the-shelf database architecture has been able to keep up with, resulting in an explosion of workaround technologies and approaches that really shouldn't need to exist.
We're now firmly in the cloud database era and can look at things like Spanner to judge how far away Postgres remains from satisfying modern availability & scaling expectations. But it will be great to see OSS close the gap (hopefully soon!).
"YugabyteDB Anywhere" is open core though right? Half the value & complexity is in the orchestration stack. It's definitely a step forward, but such licensing is likely still too restrictive to ever supplant Postgres itself (it terms of breadth of adoption).
> "YugabyteDB Anywhere" is open core though right?
No, it's all open source. YugabyteDB Anywhere is just automation & 24/7 core engineering support. All db features are open source. You can re-create Anywhere without needing to touch c/c++.
Note: All features have always been open source (besides incremental backup which will be soon).
Big databases are not new. If that was all that was needed, people would have been doing just that for the past 40 years. Turns out it doesn't always work, and sometimes (often?) it's terrible.
Even if theoretically it was all you ever needed, the other constant problem is the implementation. Most developers, today anyway, are morons. They don't understand how databases work or how to use them, so they use them poorly. In order to get away from that fact, they invented new databases (like NoSQL) so they could use their brains less and just write more glue code. Turned out that was horrible as well.
Pretty soon the tides will turn yet again and "just having a big database" will again be seen as not in vogue and another paradigm of "simplicity" will come around (instead of "one big database" being simple it'll be "many small databases" like we had with microservices).
Those who don't understand history are doomed to repeat it.
I've heard so many people claim this, but I've never seen this put into practice in production, and at scale.
I don't even think that that proves that this cannot be done... And I've seen A LOT of people try. But, things invariably start to break down, and at this point I just think that the statement above is more wishy washy than anything, though I want to believe...
How exactly is it different from No-SQL?
No schema? Check.
No consistency? Check. (eventually consistent)
Key-value store? Check. (because it's using ZooKeeper under the hood)
Promising amazing results and freeing you from the chains of the SQL? CHECK!
Everything you wrote here is false, with the exception of Rama not using SQL. Rama has strong schemas, is strongly consistent, and is not limited to key/value (PStates can be any data structure combination). Zookeeper is used only for cluster metadata and is not involved with user storage/processing in any way.
That's how things typically take off, not on the first attempt. Depends on what's different this time.
(Though NoSQL has outlived its usefulness as a concept IMO, it's just too loose to be useful beyond its early use for "something like CouchDB/Mongo", which this is clearly not)
By my reading, it's a variant of the "Kappa architecture" (aka "event sourcing").
You have a "Depot", which is an append-only log of events, and then build arbitrary views on top of it, which they call "P-States". The Rama software promises low-latency updates of these views. Applications built on this would query the views, and submit new events/commands to the Depot.
It's a backend development platform that can handle all the data ingestion, processing, indexing, and querying needs of an application, at any scale. Rather than construct your backend using a hodgepodge of databases, processing systems, queues, and schedulers, you can do everything within Rama within a single platform.
Rama runs as a cluster, and any number of applications (called "modules") are deployed onto that cluster. Deep and detailed telemetry is also built-in.
The programming model of Rama is event sourcing plus materialized views. When building a Rama application, you materialize as many indexes as you need as whatever shapes you need (different combinations of durable data structures). Indexes are materialized using a distributed dataflow API.
Since Rama is so different than anything that's existed before, that's about as good of a high-level explanation as I can do. The best resource for learning the basics is rama-demo-gallery, which contains short, end-to-end, thoroughly commented examples of applying Rama towards very different use cases (all completely scalable and fault-tolerant): https://github.com/redplanetlabs/rama-demo-gallery
What do you mean by "platform"? Is this open source? Can I run everything locally?
Is this basically an RBDMS and Kafka in one? Can I use SQL?
I understand the handwaving around programming semantics, but I'd like clearer explanations of what it actually is and how it works. Is this a big old Java app? Do you have ACID transactions? How do you handle fault tolerance?
It may be early, but I believe folks will be curious about benchmarks. And maybe, someday, Jepsen testing.
Can you please elaborate more on the open source aspect of this?
Will it be an industry revolutionizing, open-source project like containerd (Docker) that every little developer and garage-dev can built upon or will it be benefiting only the big tech corporate world that controls and benefits from power and might which will be able to pay for this?
Especially since you chose to use the name Rama, I am wondering whether this will be for the benefit of all, or only for the benefit of the few who already control more than a fair share of power(finances)?
yesn't to some degree it's the round trip back to "let's put a ton of application logic into our databases and then you mainly only need the database" times.
Just with a lot of modern technology around scaling, logging etc. which hopefully (I haven't used it yet) eliminates all the (many many) issues this approach had in the past.
not quite, through more like anything which is widely known
I have worked (a small bit) on very similar (proprietary non public internal) systems before ~5 years ago and when doing so have read block-posts about the experience some people had with similar (also proprietary internal) systems which at that point where multiple years old ...
I guess what is new is that it's something you can "just use" ;=)
I like this description. Most one point one i've seen in the thread and you doc. So its not really a tool to use, but more of framework to follow. Wouldn't be the first framework to provide tools / setup processes and workflows in a better than ever tradeoff of features/complexity/skill floor/etc.
But yeah, quite a lot of hype and red flags. My favorite from the website: "Rama is programmed entirely with a Java API – no custom languages or DSLs."
And when you look at the example BankTransferModule.java:
> .ifTrue("isSuccess", Block.localTransform("$$funds", Path.key("toUserId").nullToVal(0).term(Ops.PLUS, "*amt")))
Yeah, it's probably fair to call that a DSL, even if entirly java.
Anyway, hope to get the chance to work with event based systems one day and who knows, maybe it will be Rama.
I consider a DSL something that has its own lexer/parser, like SQL. Since Rama's dataflow API is just Java (there's also a Clojure API, btw), that means you never leave the realm of a general purpose programming language. So you can do things higher-order things like generate dataflow code dynamically, factor reusable code into normal Java functions, and so on. And all these are done without the complexity and risks of doing so by generating strings for a separate DSL, like you get when generating SQL (e.g. injection attacks).
It seems like an event sourcing database. Basically, instead of writing you write a message, then you can make read-only tables that update based on those messages. People do this today in certain domains but it is definitely more complicated than traditional databases.
More complicated in what ways specifically? I think the relevant thing is wether building an app with Rama is more or less complicated. Rama may be more complicated than mysql in implementation, but that doesn't affect me as a developer if it makes my job easier overall.
Discussing levels of complexity quickly gets pretty subjective. It is possible that Rama has found good abstractions that hide a lot of the complexity. It is also possible that taking on more complexity in this area saves you from other sorts of complexity you may encounter elsewhere in your application.
However, there is just more going on in an event sourcing model. Instead of saving data to a location and retrieving it from that location you save data to one location, read it from another location, and you need to implement some sort of linker between the two (or more).
This also comes down to my personal subjective experience. I actually really like event sourcing but I have worked on teams with these systems and I have found that the majority of people find them much harder to reason about than traditional databases.
There can be a lot of integration pain when implementing event sourcing and materialized views by combining individual tools together. However, these are all integrated in Rama, so there's nothing you have to glue yourself as a developer. For example, using the Clojure API here's how you declare a depot (an event log):
That's it, and you can make as many of those as you want. And here's how a topology (a streaming computation that materializes indexes based on depots) subscribes to that depot:
(source> my-events :> *data)
If you want to subscribe to more depots in the topology, then it's just another source> call.
That these are integrated and colocated also means the performance is excellent.
This is what has me excited about Rama. I was very into the idea of event sourcing until I realized how painful it would be to make all the tooling needed.
Actually, when done right this dramatically simplifies a backend architecture. Even a low-scale application commonly uses multiple databases (e.g. Postgres plus ElasticSearch) and queues+workers for background work.
Our Twitter-scale Mastodon implementation is a direct demonstration of this. It's literally 100x less code than Twitter had to write to build the equivalent feature-set at scale, and it's more than 40% less code than Mastodon's official implementation. This isn't because of being able to design things better with the same tooling the second time around – it's because it's built using fundamentally better abstractions.
Twitter is a simple product that only has value because of the eyeballs that look at it, not the depth of the product.
I fundamentally disagree with the premise of your blog post but not the premise of your company, so why don't I make an ask for you to write a much more practically useful application. Design a basic shopping cart using your system and compare and contrast it with a well designed relational equivalent. A system that allows products to be purchased and fulfilled is a far closer match to what the majority of companies are using software for compared to writing a Twitter clone.
Here's my take -- at a certain point in scale and volume using a database, it actually does make sense to rewrite all of the following from scratch:
- query planning
- indexing (btree vs GIN, etc) and primary/foreign/unique key structure
- persistence layer
- locking
- enforcing ACID constraints
- schema migrations/DDL
- security, accounts and permissioning
- encryption primitives
But more crucially, my belief and experience is that most companies making lots of money from software products will NEVER remotely reach that scale -- and prematurely optimizing for that is not just the wrong decision but borderline professional malpractice. You can get very far with Postgres and JSONB if you really need it, and you'll spend more time focusing on your business logic than reinventing the wheel.
I'd like to be proven wrong. But I get sinking feeling that I'm not wrong, and while your product is potentially valuable for a very specific use case, you're doing your own company a disservice by distorting reality so strongly to both yourselves and your prospective customers.
I'll round out this comment by linking another comment in this thread that goes very well into the perils of event sourcing when the juice isn't worth the squeeze:
Different applications will be more relatable to different developers. And that is the path we are going down, of steadily building up more and more examples of applying Rama towards different use cases. Some developers will get their light-bulb moment on a compare/contrast to a shopping cart, others vs. a time-series analytics app, etc. It's completely different for different developers, so building up that library of examples will take time. At the moment, we're focused on our private beta users who are technically savvy enough to be able to understand Rama through the first principles on which it's based.
We started with a Twitter demonstration because: a) its implementation at scale is extremely difficult, b) I used to work there and am intimately familiar with what they went through on the technical end, and c) the product is composed of tons of use cases which work completely differently from each other – social graph, timelines, personalized follow suggestions, trends, search, etc. A single platform able to implement such diverse use cases with optimal performance, at scale, and in a comparatively tiny amount of code is simply unprecedented.
Most of the comments here bias negative, but you shouldn't take that to heart. Hackernews is, in general, conservative when it comes to attempts to displace entrenched, battle-tested solutions (especially when they come wrapped with unfortunate, hyperbolic rhetoric).
It's a very impressive demo. You should be proud, keep your chin up, and keep us updated as Rama's value prop. continues to grow.
It's also a write-only workload that doesn't have to deal with updates intrinsic money movement, pricing, inventory, and other common workloads where atomicity is not a nice to have but a requirement. Twitter is trivial as a product compared to the most mundane shopping cart. You worked at Twitter? Okay great, I worked at AWS which runs the entire internet. And it doesn't really matter. Even if I am intimately familiar with where naive deployments of relational databases breakdown at scale, as well as mitigating approaches (cellular architecture) that make it possible to keep scaling them.
You're not really addressing the substance of my comment so I am left to assume that your omission is because you cannot address it. Be that as it may, you'll hopefully at least take my final comment at its face value that you do your own product a disservice by overhyping what its use case is towards areas that it is objectively a poor fit. People have tried event sourcing many times and it's just not a good fit for many if not most workloads. You can't be everything to everyone. There's nothing wrong with that.
My advice to you is this: call out the elephant in the room and admit that, and focus on workloads where it is a good fit. That extra honesty will go a long way in helping you build a business with sustainable differentiation and product market fit.
They didn't claim authority, they disclaimed arguing by authority: "I worked at AWS [...] And it doesn't really matter. [please address] the substance of my comment".
Look I don't have any reason to praise Twitter, but ...
This "Twitter-scale mastadon implementation" is when my red flags went up. It's meant to demonstrate a simpler and more performant architecture, but it actually demonstrates "things you should never do" #1: rewrite the code from scratch.
"The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed."
The "1M lines of code" and "~200 person-years" of Twitter being trashed on in this article is the outcome of Twitter doing the most important thing that software should do: deliver value to people. Millions of people (real people, not 100M bots) suffered thru YEARS of the fail-whale because Twitter's software gave them value.
This software has only delivered some artificial numbers in a completely made-up false comparison. Okay it's built on "fundamentally better abstractions", but until it's running for people in the real world, that's all it is: abstract.
Please don't tout this as a demonstration of how to re-create all of Twitter with simpler and more performant back-end architecture.
By your logic, all testing is invalid except for usage by real users. Two things:
- We stress-tested the hell out of our implementation well beyond Twitter-scale, including while inducing chaos (e.g. random worker kills, network partitions).
- We ran it for real when we launched with 100M bots posting 3,500 times / second at 403 average fanout. It worked flawlessly with a very snappy UX.
The second-system effect is a real thing, but there's a difference when you're building on radically better tooling. All the complexity that Twitter had to deal with that led to so much code (e.g. needing to make multiple specialized datastores from scratch) just didn't exist in our implementation.
> By your logic, all testing is invalid except for usage by real users
That is exactly correct. All testing is make believe except for real case studies by real customers with an intent to pay, and barring that, real pilots with real utilization. Otherwise, you run the risk of building a product for a version of yourself that you are pretending is other people.
The Joel article is about not rewriting an existing application. TFA is not saying that Twitter should rewrite. It's saying that if you don't work for Twitter but you want to write something like Twitter maybe this could be a good place to start.
At some point in time the same argument was made for relational databases despite there being stable systems built without them based on ISAM. The newer relational systems took a lot less work to implement but that didn't imply that it made sense to rewrite the ISAM based systems.
This is a well-known essay that everyone should read, and yet nobody should ever cite it as a commandment for what other engineers can or cannot do.
Joel was talking about commercial desktop software in the extremely competitive landscape of the 90s, he wasn't talking about world-scale internet service infrastructure. The architecture that delivered a set of X features to the initial N users isn't always going to be enough for X+Y features to the eventual 1000*N users that you promised your investors.
Companies like Google are quite public about how much they rewrite internal software, and that's just what the public hears about. A particular service might have been written with all of the care and optimization that a world-class principal engineer can manage, serve perfectly for a particular workload for several years, and yet still need to be entirely replaced to keep up with a new workload for the following several years.
You wouldn't tell another engineer that they shouldn't rewrite a single function because software should never be rewritten, so it also doesn't make sense to tell them not to rewrite an entire project either. It's their call based on the requirements and constraints that they know much more about.
Nobody should be rushing out to rewrite Linux or LLVM from scratch, and yet we wouldn't even have Linux or LLVM if their developers didn't find reasons to create them even while other projects existed. In hindsight it's clear those projects needed to be created, but at the time people would have said you should never rewrite a kernel or compiler suite.
The thing is, the abstractions you're offering is sugar on top of a well-known, well-understood old architectural pattern. The Twitter / Mastodon comparison is unconvincing exactly because both Twitter and Mastodon seemingly totally ignored the well-known, well-trodden ground on this, for no good reason.
If you can convince more developers to apply what is effectively canonical store -> workers -> partitioned denormalized materialized views as a pattern where it makes sense, then great.
But you can do that with just the tools people already have available. Heck, you can do that with just multiple postgres servers (as the depot, and for the "p-stores" and for the indexing functions), and then you don't need to ditch the languages people are familiar with for both specifying the materialization and the queries.
Part of the reason we use a "hodgepodge of narrow tooling" however, tends to be that it allows us to pick and choose languages, and APIs, depending on what developers are familiar with and what suits us, and it allows people to pick and mix. Convincing people to give up that in favour of a fairly arcane-looking API restricted to the JVM is going to be a tough sell a lot of places.
Mastodon is extremely similar to the Twitter consumer product. The differences are minor. We compare with Twitter because unlike Mastodon Twitter runs at large scale.
Yes, we implemented the entirety of Mastodon from scratch.
Well, if your program domain requires X level of essential complexity, you will have to achieve that somehow. You can either let it live in someone else's code, that's already written and battle tested, or you can rewrite it yourself, which is time consuming and the results might be subpar. There is no free lunch.
Complexity can't be destroyed, only moved around. For a great number of tasks this solution offers a bad set of trade offs.
But, at a certain level of scale everything is a data engineering problem, and sometimes this is the (relatively) simple solution when viewed in the context of the entire system.
'Just use mySQL/SQLite/Postgres' is great advice until it isn't.
Every problem has an amount of essential complexity, which as you said can’t be done away with. Imperfect engineering often adds unnecessary (accidental) complexity.
I think it would a mistake to approach a domain and assume nothing can be made simpler or more straightforward.
In all the companies I've worked at "event sourcing plus materialized views." has only ever caused intense confusion and resulted in more bugs and more downtime. The simpler solutions of either using MySQL or PostGres OR Redis/DynamoDB, all have worked better.
I genuinely believe we would all be better off if Martin Fowler's original article about Event Sourcing had never been written. IMO, it's a bad idea in 99% of cases.
I've worked with people who did work with successfully employed event sources architectures.
One thing there is: These were massive engineering departments in very large companies. Think of several dozen data processing teams, each very much their own small company, all working on different domains with event and stream consumers and utilizing the possibility of restarting their stream consumers some time ago. Impressive Kafka-Clusters ingesting thousands and thousands of events per second and keeping quite the backlog around to enable this refeeding, outages and catching back up and such. At their scale and complexity, I can see benefits.
However, at that scale, Kafka is your database. And you end up with a different color of the same maintenance work you'd have with your central database. Data ends up not being written, data ends up being written incorrectly, incorrectly written data now ends up with transitive errors. At times, those guys ended up having to drop a significant timespan of data, filter our the true input events, pull in the true events and start refeeding from there - except then there was a thundering herd effect, and then they had to start slowly bringing up load... great fun for the management teams of the persistence.
Note that I'm not necessarily saying e.g. Postgres is the solution to everything. However, a decently tuned and sized, competently run Postgres cluster allows you to delay larger architectural decisions for a long time with run-of-the-mill libraries and setup. Forever in a nonzero amount of projects.
This is more an advertisement for a type of database than a statement that they are unnecessary.
From what I can tell in the article it seems their differentiator is Event Sourcing and having arbitrary complex index builders on top of the events. It seems similar to EventStoreDB[1].
I have always been interested by the concept of an event sourcing database with projections and I want to build one eventually so it is interesting to see how they have approached the problem.
Also they mention on their site:
> Rama is programmed entirely with a Java API – no custom languages or DSLs.
It makes sense why they have gone this route if they want a "Turing-complete dataflow API" but this can be a major barrier to adoption. This is a big challenge with implementing these databases in my opinion because you want to allow for any logic to build out your indexes/projections/views but then you are stuck between a new complicated DSL or using a particular language.
True, Rama does have a learning curve. It's easier to explain why its indexing capabilities are so powerful, as data structures vs. data models is fairly relatable to most devs. But I actually consider its dataflow API to be the bigger breakthrough. Our Twitter-scale Mastodon implementation and examples in rama-demo-gallery are demonstrations of this.
Also, Rama has a first-class Clojure API (I should probably update the website).
I get your point about global mutable state, but I do not see how you have done anything more than shuffle the complexity around
Being dependant on software like the JVM makes me very reluctant to investigate further. The JVM is not involved in anything I use, I do not think i want it.
> This is more an advertisement for a type of database than a statement that they are unnecessary.
That was my reaction as well. The article's claimed argument against databases is that they are global mutable state, which is supposed to be bad. But none of what the article advocates for gets rid of global mutable state (which of course you can't do in any real world application, because the real world itself is global mutable state and anything that deals with the real world has to deal with that). It's just hawking a particular way of storing global mutable state.
T.b.h. when looking at things, global mutable state might not be bad, if it is principled?
I.e. I did like how Clojure had it's atomically updating refs.
And you realisticaly won't get more principled than with traditional RDBS, with it's ability to transactions, rollbacks, e.t.c.
Only alternative that I have seen proposed to using DBAS while not relaxing the guaranties, is to double down on event-sourcing and not having mutable-state anymore. Everything could be just a fold over all of the incomming events, right? But I don't think I have seen non-toy example of that anywhere I worked at.
That's not really what people mean by immutable though. What they mean is something halfway between the casual definitions of "immutable" and "persistent," in that your copy of the data at point X will always be valid, even if new data comes in later.
> That's not really what people mean by immutable though.
As far as the term "global mutable state" is concerned, which is what the article claims is bad, the event store is global mutable state. One can think of the individual event records in the store as immutable (they don't change once they're recorded), but the article is talking about the entire store, not an individual record in it.
> your copy of the data at point X will always be valid
If your copy of the event store is out of date (missing some events), it's not valid. A new event that your copy hasn't yet registered might change the results of queries on the store, meaning that your copy will be delivering out of date, hence invalid, query results. You can't handwave around that by gerrymandering the definition of "immutable". The simple fact is, as I said, that the real world is global mutable state, and if your app is dealing with the real world, it has to deal with that.
Append-only log-based data-stores are close enough in my book.
And conceptually, you can then push all of those "what if I am missing events, or tey are in wrong order" into the fold over the data.
Because real world is mutable data, but history of the world at any particular point isn't. It can be incomplete, it can be a personal view, I like that there are things made explicit that otherwise wouldn't. Nice abstraction.
Like, if you are splitting hairs, is even array-append in Haskell immutable? It does create new data somewhere in the memory. But I don't think that makes Okasaki's thesis and book any less valuable :)
> Append-only log-based data-stores are close enough in my book.
Close enough to what? I get that there are advantages to storing global mutable state this way. I just don't see the point of denying that what you are doing is storing global mutable state--every time you append a new log entry to the data store, you mutate the state.
> history of the world at any particular point isn't
Sure, I get that too, and I get that it's nice to have that stored explicitly instead of having to reconstruct it for the particular cases where you need it.
But if someone is running a query on your database, except in some particular cases, they don't want the history of the world at some particular point to dictate the response. They want the most up to date history of the world to dictate the response. And that changes every time you append a new log entry to the data store. Again, I don't see the point of denying that that's what is going on.
> is even array-append in Haskell immutable?
The point of "immutable" in Haskell is that every object is immutable. So array-append in Haskell does not take an existing array object and append an element to it (as the corresponding operation would for, say, a list in Python). It constructs a new array object that is the same as the old object with the new element appended. So the old array object is not mutated.
But the array-append operation of course does change the global state of the program, as you say. So there is no such thing as an immutable global state of a Haskell program. And of course the same would be true for a program in any language.
> I don't think that makes Okasaki's thesis and book any less valuable
Of course it doesn't. But it also doesn't mean the definition of "immutable" that he is using is the only possible or useful one. The article under discussion here is using a different definition, which for the topic it is discussing is more useful. Once more, I don't see the point of trying to twist words around to deny that.
The article calls out event-sourcing and materialized views as a solution.
I reread the article twice now, and I don't see a conflicting definition, that would contradict i.e. append-only-log being immutable.
Unless your argument is that the author is in-denial about benefits of immutability, because it can't be achieved in this weirdly strict form you seem to propose?
But maybe I need to reread second paragraph of the article one more time?
> The article calls out event-sourcing and materialized views as a solution.
Yes, I know, and it may well be a good solution to various issues with conventional databases. I just don't see how it's a solution to global mutable state, since there still is global mutable state.
> I don't see a conflicting definition
The article never gives an explicit definition of "global mutable state". It also does not explicitly say that its proposed solution gets rid of global mutable state. The section "Global mutable state is harmful" strongly implies that its proposed solution will somehow get rid of global mutable state, but the article never actually says that it does. It just goes into a bunch of other benefits of event sourcing and materialized views, all of which might well be valid benefits, but none of which amounts to getting rid of global mutable state.
> Unless your argument is that the author is in-denial about benefits of immutability
I can't see into the author's mind, so I don't know if the aspects of the article that I described above were inadvertent, because the author simply hasn't really thought through what "global mutable state" actually means, or whether it's on purpose, to imply a claim about the article's proposed solution that sounds good but isn't actually true, as a marketing tactic.
As I said in my earlier response, the article doesn't explicitly define what "global mutable state" is, but it does say that conventional databases are global mutable state the same way that global variables in an ordinary program are.
By that definition, an append-only log as a "single point of truth" data store, which is the article's proposed "solution", is also global mutable state. The article does not acknowledge this, but it seems obvious to me.
Yes, there is. Appending a new log entry to the depot mutates the depot. It doesn't mutate earlier records in the depot, but the depot is the entire record set, not just individual records. The record set is different after the append than before it.
Good point. I don’t use red planet labs product myself either, but in a world where it is hard to get people to even consider your product the article did work
A Java API doesn't mean you can only access it from Java. Such APIs can also be accessed from most languages. Obviously bytecode languages like Clojure, Kotlin, Scala, Groovy etc. But also via Truffle languages like JS, Python, Ruby. All without bindings.
You can also do C++ and Rust, but that does require a bit of binding work.
So it's not really limiting you to a single language, or these days, even a single runtime.
Assuming you're willing to run a JVM and possibly specific language implementations. That's fine for a lot of people, I'm sure, but it's not something I'd be willing to do for what is fairly little benefit over "just" whatever solution you're already familiar with to use as the depot and the "P-stores" with workers in between.
Yeah, the Java dependency makes me reject it instantly - the only advantage they potentially could have over "just" using any database with ability to query remote databases is not having to roll your own, but the barrier to rolling my own is lower for me than introducing a Java dependency.
(A replicated set of databases for the depots; a set of workers creating the "p-stores" in ... another set of databases, and you've replicated the architecture; heck you can even run this entirely off postgres w/replication + triggers to handle partitioning and materialization with no external workers if you like)
EDIT: Also their API fairly unambiguously is effectively a DSL, just one without it's own separate syntax and parser allowing access from outside the JVM.
I've had enough problems with JVM deployments over the years to just not want the hassle in order to use something which adds marginal value. If you don't, then by all means, use it - I'm not suggesting it's bad for everyone. If it provided some massive major advantage over the large number of other options, I'd consider it too, but I don't see anything that indicates that's the case here.
Do you reject Kafka because it runs on the JVM? This is a ludicrous position to take. You may not want to fund development of projects when you're forced to train devs to work them, but to reject using technologies based on the programming language / framework is just fundamentally flawed. Don't use Python tools because I hate python. Don't use kubernetes because it's golang don't use AWS because it has a bunch of java services. S3? I'm not entirely sure but I wouldn't be surprised if it was java edge service based as well.
I need to use Java exactly zero times when interacting with those, they have dedicated clients for everything, something this stuff apparently is proud of not having.
> If it provided some massive major advantage over the large number of other options, I'd consider it
People reject using technologies because of the impact of having to support them all the time when the benefit does not justify needing to support one more technology. I don't have an issue with using JVM based services if I can have someone else manage them for me, but when I have to deal with the operational issues myself, or within my team, then how comfortable we are with managing a specific technology vs. the advantages it brings over competing alternatives absolutely is part of the consideration, and it'd be irresponsible for it not to be.
In this case the benefit seems marginal at best, and I'm not deploying a JVM based service for marginal benefits given how much hassle they've brought me over the years.
Nothing is outright rejected because it runs on the JVM, but it is a detracting quality, adding complexity. If Kafka is clearly the best solution available for your problem, you'll suck it up and the use JVM, but if there is another approach that will also serve your needs without a JVM dependence then the choice is easy.
Well, it is an advertisement, but much of what it says is undoubtedly true. It's true that databases are effectively global mutable state in practice, with all the headaches that gives, and that event sourcing and materialized views is a damn appealing solution to it.
However, the jump from that to saying you have it is a bit like the jump from talking about the messiah and saying that he's sitting in the next room just now.
We investigated event sourcing and materialized views as the backbone of a (new) big business application five years ago. The problem was that there were so few products back then and they were so untested (unless my mind is playing tricks on me, we did look at EventStoreDB, as well as Kafka streams), and despite being open source they were also usually heavily tied to one commercial provider. No one had anything like experience with it. I couldn't convince my colleagues to go for this, much less my bosses. Hell, I could barely convince myself.
Philosophically, I believe it's sound, but how much has changed in those years?
> This is more an advertisement for a type of database than a statement that they are unnecessary.
I stopped reading the article once it was evident that the author was not making any case regarding databases being unnecessary, and instead was putting up a series of strawmen that sounded like desperate attempts to reject all alternatives to the thing they were selling.
The author's attempt to talk about data structures was particularly baffling, knowing that pretty standard RDBMS such as PostgreSQL already support JSON data types. Are we supposed to forget about that?
That's kind of the point. Model your data. Think about it. Don't (mis)treat your database as a "persistence layer" -- it's not. It's a knowledge base. The "restriction" in the relational model is making you think about knowledge, facts, data, and then structure them in a way that is then more universal and less restrictive for the future.
Relations are very expressive and done right is far more flexible than the others named there. That was Codd's entire point:
"Future users of large data banks must be protected from
having to know how the data is organized in the machine (the
internal representation) ..." and then goes on to explain how the predicate-logic based relational data model is a more universal and flexible model that protects users/developers from the static impositions of tree-structured/network structure models.
All the other stuff in this article is getting stuck in the technical minutiae of how SQL RDBMSs are implemented (author seems obsessed with indexes). But that's somewhat beside the point. A purely relational database that jettisons SQL doesn't have to have the limitations the author is poking at.
It's so frustrating we're still going over this stuff decades later. This was a painful read. People developing databases should already be schooled in this stuff.
Precisely this. Every “solution” I’ve seen approaches RDBMS as an obviously incorrect kludge that must be done away with in the name of DX.
RDBMS hasn’t stuck around this long only because it’s good enough. It’s an incredibly powerful way to model data, for now and the future, but yes, it does require you to think carefully and slow down. These are not things to be avoided.
Yes, the core of the problem with people "getting it" has always been that SQL has been so mediocre, and yet it's the only serious option people have.
And so people go looking for something with a modern syntax, that's properly composable, functional, etc. but because they haven't studied the foundations, they reinvent the wheel badly.
The Codd paper is so readable, and so persuasive. People need to start there, and then argue why their novel approach is better on first principles, not just because it smells better than SQL.
But that it's stuck around and done so well for so long tells you that the core foundation is sound.
At the beginning of the article, the "relational" database is described as a particular architecture ("map of maps, with secondary indexes being additional maps") without recognizing that the other given examples ("key/value", "document" and "column-oriented") are also relational and the differences are only, at worst, performance tradeoffs.
If Postgres was already horizontally scalable and supported incrementally maintained recursive CTEs (like Materialize can do) then I could see how Rama would be mostly uninteresting to a seasoned SQL developer, but as it is I think Rama is offering a pretty novel & valuable set of 3GL capabilities for developers who need to build scalable reactive applications as quickly as possible. Capabilities which other SQL databases will struggle to match without also dropping down to 3GL APIs.
> A purely relational database that jettisons SQL doesn't have to have the limitations the author is poking at.
Agreed. Relational databases can take us a lot further yet.
In their chosen example, and many others that involves external sources, it's not really "your" data, though. You have no control over the structure, and you then have the choice of deciding whether you want to throw away what you can't shoehorn into your current schema, or store it and transform it as best you can and be prepared to re-transform it as your system evolves.
Really, they're not avoiding modelling the data. They're modelling it in their "P-stores" whether they want to admit that's just another data model or not. It's clearly not data models they object to, but typical database DDL coupled with a desired to not throw away data that doesn't conform right now. Depending on your application domain, not throwing away data that doesn't conform to your current data model can be good or bad, but it's not an accident that they picked Twitter/Mastodon where it's easy to justify not rejecting non-conforming documents.
I agree with you that this doesn't require ditching relational models, though. For that matter not even SQL. You can even fairly easily build this kind of architecture without e.g. ever leaving Postgres (replicated pair for their "depots"; triggers + postgres foreign data wrappers to materialize views on index servers and/or to do requests across index servers).
We do go in circles/cycles quite a lot as an industry. I wonder if the trend is back towards SQL, right now, too many teams been burned by Event Sourcing when they just needed a decent SQL DB? Just idle conjecture....
Is this Rama solution similar to the kind of thing you can get with Kafka with KTables?
If so I'd be curious how they've solved making it in anyway operational less complex to manage then a database. It's been a few years since I've run Kafka but it used to kind of be a beast.
Even after reading this doc [1], I am not clear on who is the target audience and what are you trying to solve? It would be helpful to take a real world example and translate how easy /efficient it would be to do this via RAMA.
The first is our Twitter-scale Mastodon implementation, which is 100x less code than Twitter wrote to build the equivalent at scale (just the consumer product). It's also more than 40% less code than Mastodon's official implementation (which isn't scalable). https://github.com/redplanetlabs/twitter-scale-mastodon
The rama-demo-gallery repo also contains many short, self-contained, thoroughly commented examples of applying Rama towards very different use cases. These include user profile management, time-series analytics, atomic and fault-tolerant bank transfers, and more. https://github.com/redplanetlabs/rama-demo-gallery
Mastodon is extremely similar to the consumer Twitter product, and "Twitter-scale" is a known number that we tested well beyond. Among other things, we verified our timeline delivery latency was at least as good as Twitter's number ad that it scaled linearly.
"Global mutable state is harmful" - well... yes, that's totally correct. "The better approach [..] is event sourcing plus materialized views." .....errr... that's one approach. we probably shouldn't hitch all our ponies to one post.
"Data models are restrictive" - well, yes, but that's not necessarily a bad thing, it's just "a thing". "If you can specify your indexes in terms of the simpler primitive of data structures, then your datastore can express any data model. Additionally, it can express infinite more by composing data structures in different ways" - perhaps the reader can see where this is a bad idea? by allowing infinite data structures, we now have infinite complexity. great. so rather than 4 restrictive data models, we'll have 10,000.
"There’s a fundamental tension between being a source of truth versus being an indexed store that answers queries quickly. The traditional RDBMS architecture conflates these two concepts into the same datastore." - well, the problem with looking at it this way is, there is no truth. if you give any system enough time to operate, grow and change, eventually the information that was "the truth" eventually receives information back from something that was "indexing" the truth. "truth" is relative. "The solution is to treat these two concepts separately. One subsystem should be used for representing the source of truth, and another should be used for materializing any number of indexed stores off of that source of truth." this will fail eventually when your source of truth isn't as truthy as you'd like it to be.
"The restrictiveness of database schemas forces you to twist your application to fit the database in undesirable ways." - it's a tool. it's not going to do everything you want, exactly as you want. the tradeoff is that it does one thing really specifically and well.
"The a la carte model exists because the software industry has operated without a cohesive model for constructing end-to-end application backends." - but right there you're conceding that there has to be a "backend" and "frontend" to software design. your models are restrictive because your paradigms are. "When you use tooling that is built under a truly cohesive model, the complexities of the a la carte model melt away, the opportunity for abstraction, automation, and reuse skyrockets, and the cost of software development drastically decreases." - but actually it's the opposite: a "cohesive model" just means "really opinionated". a-la-carte is actually a significant improvement over cohesion when it is simple and loosely-coupled. there will always be necessary complexity, but it can be managed easier when individual components maintain their own cohesion, and outside of those components, maintain an extremely simple, easy interface. that is what makes for more composable systems that are easier to think about, not cohesion between all of the components!
"A cohesive model for building application backends" - some really good thoughts in the article, but ultimately "cohesion" between system components is not going to win out over individual components that maintain their cohesion and join via loosely-coupled interfaces. if you don't believe me, look at the whole Internet.
Every time I tried to use event sourcing I have regretted it, outside of some narrow and focused use cases.
In theory ES is brilliant and offers a lot of great functionality like replaying history to find bugs, going back to any arbitrary point in history, being able to restore just from the event log, diverse and use case tailored projections, scalability, ...
In practice it increases the complexity to the point were it's a pointless chore.
Problems:
* the need for events, aggregates and projections increases the boilerplate tremendously. You end up with lots of types and related code representing the same thing. Adding a single field can lead to a 200+ LOC diff
* a simple thing like having a unique index becomes a complex architectural decision and problem ... do you have an in-memory aggregate? That doesn't scale. Do you use a projection with an external database? well, how do you keep that change ACID? etc
* you need to keep support for old event versions forever, and either need code to cast older event versions into newer ones, or have a event log rewrite flow that removes old events before you can remove them from code
* if you have bugs in you can end up needing fixup events / event types that only exist to clean up , and as above, you have to keep that around for a long time
* similarly, bugs in projection code can mess up the target databases and require cumbersome cleanup / rebuilding the whole projection
* regulation like GDPR requires deleting user data, but often you can't / don't want to just delete everything, so you need an anonimizing rewrite flow. it can also become quite hard to figure out where the data actually is
* the majority of use cases will make little to no use of the actual benefits
A lot of the above could be fixed with proper tooling. A powerful ES database that handles event schemas, schema migrations, projections, indexes, etc, maybe with a declarative system that also allows providing custom code where necessary.
It's a way of doing stream processing that sacrifices a little bit of update latency for higher throughput and exactly-once processing semantics. Regular streaming has single-digit milli update latency, while microbatching has at least a few hundred millis update latency.
By "exactly-once processing semantics", I mean that regardless of how many failures are on the cluster (e.g. nodes losing power, network partitions), the updates into all partitions of all indexes will be as if there were no failures at all. Pure streaming, on the other hand, has either "at-least once" or "at-most once" semantics.
This is marketing spiel masquerading as a bad take. Rama may or may not be cool tech, but the idea that they are anywhere close to being able to get rid of structured database systems for complex systems is absolutely laughable to the point that it makes me uninterested in learning more about the tech. Please tone down the hyperbole if you want serious attention.
My biggest problem with databases is that they are very hard to evolve. They accumulate a history of decisions and are in a suboptimal state. Legacy is widespread in enterprises. Oracle is still milking $ 50B+ annually, and the databases are the primary driver of why you need Oracle and why they can upsell you other products after a compliance audit.
The schema changes are hard (e.g. try to normalize/denormalize data), production is the only environment when things go wrong, in-place changes with untested revert options are default, etc.
Clearly, there's some truth to this perspective. On the other hand, which alternative actually makes this easier? If your schema is non-enforced, and/or your data (and its schema) are in loosely coupled distributed silos, and/or as in the articles case indexes are essentially little programs... Then broad scale schema evolution goes from being difficult to often being effectively entirely infeasible.
It's possible that in turn forces users to adapt and split their changes into smaller chunks, but that's pretty speculative and they might simply fail entirely in the attempt.
Which data stores are more easily evolved than a relational DB with a fairly restrictive schema?
I agree with you that schema has tremendous value. Let's store JSON everywhere and infer it on read, backfires in a big way. Even MongoDB added tooling to enforce a schema.
I just wish there was a better tooling to evolve schema. Blue/green deployments, schema as code (no manual up and down procedures), good backward compatibility story, shadow testing in production, etc.
The database still feels like a monolith and did not get the benefits of the microservice revolution.
A lot of the commenters seem like database fans instinctively jumping to defend databases. The post is talking about contexts where you are dealing with petabytes of data. Building processing systems for petabytes has a separate set of problems from what most people have experienced. Having a single Postgres for your startup is probably fine, that's not the point here.
There is no option to just "put it all in a database". You need to compose a number of different systems. You use your individual databases as indexes, not as primary storage, and the primary storage is probably S3. The post is interesting and the author has been working on this stuff for a while. He wrote Apache Storm and used to promote some of these concepts as the "Lambda architecture" though I haven't seen that term in a while.
I think the complexity gets really crazy at high scale, but the complexity caused by databases is still significant at low scale as well. For example, needing to use an ORM and dealing with all the ways that can leak is pure complexity caused by not being able to index your domain model directly.
* The immutability, lambda architecture points I agree with. I think the separation of the immutable log from the views is important. Databases are frequently used in ways that go against these principles.
* I am not sold that being unable to express the domain model correctly is really a fair criticism of databases. Most businesses in my experience have a domain that is modeled pretty well in a relational DB. I haven't seen a better general solution yet, though I haven't checked out Rama.
At the low end of the scale, there are a lot of companies (or projects) for which the entire dataset fits in a single managed Postgres instance, without any DBA or scalability needs. They still suffer from complexity due to mutable state, but the architectural separation of source of truth vs "views" can be implemented inside the one database, using an append only table and materialized views. There are some kinds of data that are poorly modeled this way (e.g images) but many kinds that work well.
So I don't really view the architectural ideas as repudiating databases in general, more as repudiating a mutable approach to data management.
So what you're saying is that this article is irrelevant for 99.999% of developers. The instinctive jump to defend databases is completely understandable given that context.
> You use your individual databases as indexes, not as primary storage, and the primary storage is probably S3.
Which is a perfectly valid use for a database. Our company's document management system uses a big database for metadata and then, of course, stores the actual files on disk.
> However, storing data normalized can increase the work to perform queries by requiring more joins. Oftentimes, that extra work is so much you’re forced to denormalize the database to improve performance.
Databases have materialized views though, that solves this problem.
Did I miss something, or does that post completely omit concepts like concurrency, isolation, constraints and such? And are they really suggesting "query topologies" (which seem very non-declarative and essentially making query planning/optimization responsibility of the person writing them) are a superior developer environment?
This stuff is all covered thoroughly in our docs. This is a post about the complexity aspect of backend development and how those are addressed with Rama. It's not a thorough explanation of every aspect of Rama, since that would be extremely long. If you dig into Rama, you'll see that it's properties and guarantees are very strong.
And yes, Rama's queries are a massively superior approach. The need for complex query planners is a result of limitations in how you index data, oftentimes from the tension between normalization and denormalization. With Rama, it's easy to robustly materialize multiple views that are already in the shape needed for queries.
I did check the docs. According to the search, there are like 5 references to "consistency", 4 of which are talking about how traditional databases do that poorly and the 5th one seems to suggest to use "depot partitioner" which seems very much like sharding with per-shard consistency. For "isolation" there are t2 references, for "transaction" 1, none of which explains anything.
And I'm sorry, you won't convince me something defined in Java is superior to declarative SQL. There's a lot of problems with SQL, no doubt about it.
Seems like a bunch of buzzwords and such. I've been working with databases for years for one of the largest companies in the world and no one has ever said "topology" before.
Any time I would save with this is wasted on learning java and this framework.
Our production-ready, Twitter-scale Mastodon implementation in 100x less code than Twitter wrote to build the equivalent feature-set (just the consumer product) begs to differ that it's "a bunch of buzzwords". https://github.com/redplanetlabs/twitter-scale-mastodon
Meh,its better than nothing, but real world traffic is often very different than simulated traffic. If it isn't actually working with real traffic than i am not impressed.
> It’s common to instead use adapter libraries that map a domain representation to a database representation, such as ORMs. However, such an abstraction frequently leaks and causes issues. ...
FWIW, I'm creating a tool (strategy) that is neither an ORM or an abstraction layer (eg JOOQ) or template-based (eg myBatis). Just type safe adapters for normal SQL statements.
Will be announcing an alpha release "Any Week Now".
If anyone has an idea for how to monetize yet another database client library, I'm all ears. I just need to eat, pay rent, and buy dog kibble.
>If anyone has an idea for how to monetize yet another database client library, I'm all ears. I just need to eat, pay rent, and buy dog kibble
FWIW jOOQ's model worked great from a consumer/end-user point of view. I got to learn the whole library with no real strings attached, and then pay to adapt it/support it on Big Fucking Databases(TM) as projects required it. Their site and documentation is also quite pleasant.
In a way it feels like double-piggybacking: Microsoft, Oracle, et al. spend a lot on mindshare to make sure technical leads choose those respective stacks. Then the army of developers/consultants, required to get the promised ROI out of those stacks, inevitably have to go looking for tools. Be there to sell them the shovels. It helps if they are already familiar with it. (Free OSS adapters / "community" editions, etc., or even just very generous "evaluation" versions with no real effort to police use in a lab environment.)
Not sure how successful jOOQ has been financially, but considering they've been around for many years at this point, I have to imagine it's worked out well enough to pay for the lights and kibble?
The only other "dev tool" company I've enjoyed working with _more_ has been JetBrains. (The recent AI plugin not-withstanding.)
The domain knowledge embodied in JOOQ is kind of daunting. Truly huge.
Of the current SQL client API tools, mine is most like sqlc. Mine's major improvement, IMHO of course; but concept and workflow are comparable.
What do you think of paying to use such a tool for builds? eg Deployed to a CI/CD pipeline.
Free to use for sql-fiddle, personal, desktop, FOSS, etc. But once a project starts to use a tool for "real work", that's proof of value. Right?
Part of my just wants to want to do shareware. I published shareware in the early 1990s. Since I have the biz acumen of a sea cucumber, I just did the work and let people decide. It was great gig while it lasted.
I guess today's equiv would be some kind of patreon. My reluctance there is the PR/promotion/social media part. I have trouble even imagining what that'd look like.
> Not sure how successful jOOQ has been financially, but considering they've been around for many years at this point, I have to imagine it's worked out well enough to pay for the lights and kibble?
Pretty interesting once you read past the marketing push.
I mostly like the approach, but there are a lot of questions/issues that spring to mind (not that some of them don't already have answers, but I didn't read everything). I'll list some of them:
* I'm pretty sure restrictive schemas are a feature not a bug, but I suppose you can add your own in your ETL "microbatch streaming" implementation (if I'm reading this right, this is where you transform the events/data that have been recorded to the indexed form your app wants to query). So you could, e.g., filter out any data with invalid schema, and/or record and error about the invalid data, etc. A pain, though, for it to be a separate thing to implement.
* I'm not that excited to have my data source and objects/entities be Java.
* The Rama business model and sustainability story seem like big question marks that would have to have strong, long-lasting answers/guarantees before anyone should invest too much in this. This is pretty different and sits at a fundamental level of abstraction. If you built on this for years (or decades) and then something happened you could be in serious trouble.
* Hosting/deployment/resources-needed is unclear (to me, anyway)
* Quibble on "Data models are restrictive": common databases are pretty flexible these days, supporting different models well.
* I'm thinking a lot of apps won't get too much value from keeping their events around forever, so that becomes a kind of anchor around the neck, a cost that apps using Rama have to pay whether they really want it or not. I have questions about how that scales over time. E.g., say my has depot has 20B events and I want to add an index to a p-state or a new value to an enum... do I need to ETL 20 billion events to do routine changes/additions? And obviously schema changes get a lot more complicated than that. I get that you could have granular pstates but then I start worrying about the distributed nature of this. I guess you would generally do migrations by creating new pstates with the new structure, take as much time as you need to populate them, then cut over as gradually as you need, and then retire the old pstates on whatever timeline you want.... But that's a lot of work you want to avoid doing routinely, I'd think.
I'm starting to think of more things, but I better stop (my build finished long ago!)
* By "restrictive schemas" I mean being forced to represent your data storage in non-optimal ways – like not being able to have nested objects in a first-class way. Schemas themselves are extremely important, and they should be as tight as possible.
* Rama's JVM-based, so the entire ecosystem is available to you. You can represent data as primitive types, Java objects, Protobuf, Clojure records, etc.
* You deploy and manage your own Rama clusters. The number of nodes / instance types depends on the app, but Rama doesn't use more resources than traditional architectures combining multiple tools.
* Some databases support multiple very specific data models (e.g. Redis). I don't consider that flexible compared to Rama, which allows for arbitrary combinations of arbitrarily sized data structures of arbitrary partitioning.
* Depots (the "event sourcing" part of Rama) can be optionally trimmed. So you can configure it to only keep the last 2M entries per partition, for example. Some applications need this, while others don't.
* If you're adding a new PState, it's up to you how far back in the depot to start. For example, you could say "start from events appended after a specific timestamp" or "start from 10M records ago on each partition".
* We have a first-class PState migrations feature coming very soon. These migrations are lazy, so there's no downime. Basically you can specify a migration function at any level of your PStates, and the migration functions are run on read. In the background, it migrates iterates over the PState to migrate every value on disk (throttled so as not to use too many resources).
362 comments
[ 1.8 ms ] story [ 274 ms ] threadIn theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of things and their relations.
Practically speaking, the scope of a given database/schema is generally restricted to one business or problem area, but even this doesn't matter as long as the types aren't aliasing inappropriately. You could put a web retailer and an insurance company in the same schema and it would totally work if you are careful with naming things.
Putting everything into exactly one database is a superpower. The #1 reason I push for this is to avoid the need to conduct distributed transactions across multiple datastores. If all business happens in one transactional system, your semantics are dramatically simplified.
> Putting everything into exactly one database is a superpower.
Especially this.
$1M big iron DB server is much cheaper than a redundant array of inexpensive databases when people come to actually use the data; be it developers, analysts or leadership, everybody saves time, except perhaps a few DBAs.
in all seriousness, putting the data in one place also makes it a lot easier to optimize and to amortize the optimization efforts.
There were challenges like the difficulty of working with a few huge tables that existed since the company's inception and were practically infeasible to do any schema changes on, but that's more a modeling issue. It was also a single point of failure service. If the db goes down so to does everything but constant sense of impending doom it was less of a problem than I thought with ndb.
There are so many crazy tricks out there. You would be amazed at how much abuse an old school RDBMS engine can take on modern hardware. In-memory OLTP on servers that can host terabytes of RAM completely changes the game. Disaggregated compute & storage provides seamless OLAP capabilities on the same SQL connection instance for the most modern providers.
Really, as it pertains to this, it is a question of "where do I want to move the complexity to?", not "can I eliminate it?"
But yeah.
https://en.m.wikipedia.org/wiki/Sabre_Corporation
Those things were completely different from what you get in a modern database. Your example is closer to AWS than to Postgres.
It’s also a lot easier for me to handle tenancy/isolation/different data “realms” this way. And I sleep better at night knowing a mistake (I have no dbas, and I don’t have time to invest in tons of safeguards as a solo dev when there are much higher EV tasks) won’t nuke absolutely everything.
With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries.
If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.
It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
I've met plenty of pretty senior people believing the opposite and most of them likely aren't good liars.
When the dumbshit over engineered "cloud architecture" that was pushed on you (managers don't get promoted or switch companies for a big raise by reducing their headcount... no, they get promoted by building a big team for a successful [please don't check in on how it was doing two years later...] "cloud transformation") has all kinds of problems and the costs shoot into the stratosphere, while five servers on a rack would have granted greater uptime and performance in practice for your usage patterns, and lower costs... it's not your fault.
Have some bad luck and that simple solution you pushed for happens to experience an unlikely failure causing a significant outage in year 1, and your head's on the chopping block.
If everything will be my fault in the end, then screw it - I will go ahead and take ownership/control of all of those things and make sure they don't come back around to bite me in the ass. If someone questions my authority, I always cast it in terms of "how many weekends with your family are you willing to sacrifice to support that idea?" Oftentimes this is all it takes to get those idle suggestions put in the bin so we can move forward.
Solving database circus in a real business environment requires disengaging all of the safeties and getting very messy with your free time, stress levels and other humans. It's a front-loaded suffering though. If you actually know what you are doing and get through the schema refactor (and it sticks), everything is down-hill from that point. Getting one schema to rule them all and then having the team learn that schema == new common language by which you can communicate that did not exist prior. In our business, non-developers are just as aware of our tables, columns and relations as developers & DBAs are. In some cases, they have more detail about how a field is used than I do.
use the appropriate tools to solve the problem at hand. it depends.
only if the people who are hiring you are just as ignorant
After 20 years I am actually getting a bit skeptical behind the "you should constantly be learning" mantra. If you are constantly learning new, tech, you will never be an expert in anything. (That statement is very much a generalization and I am sure there are tons of people that will point out issues with it).
While I’m pre-launch and doing testing, using a more traditional vertical model is money out of my pocket. And it’s not something I want to maintain after I launch either.
Also, I’m creating something (a freemium game) where as a solo dev it will only be worth it for me to continue working on the business if it sees explosive growth. If I’m lucky enough for that to materialize, there will be at least a few months before I can hire fulltime help for the product (while also being busy doing tons of other stuff) - it would be terrible to have to handle tons of db operations tasks or re-architect things to support growth during that time.
Basically, vertical scaling is optimizing for the wrong level of usage. It’s actually the same for many of the snarky “your startup doesn’t need horizontal scaling” comments here - if your states are {building, huge growth, failure} then horizontal scaling is perfect because it keeps your costs low while you’re building and handles the huge growth phase more gracefully. Yes, maybe you’ll never get to the huge growth phase, but the entire business is oriented around either getting there or failing.
But the vast majority of small companies are not like that. They acquire one customer, then another, and another, and grow N<=50% per year, with tons of headroom on default app configurations. For those companies, worrying about explosive, global scaling instead of, you know, building their business, is time they can't get back.
We had the web app set up on Kubernetes, yet the background jobs had to be manually managed by admins at the company. There were less than 300 users in total and the only time Kubernetes scaled beyond one pod was when something was broken. When I joined, they had just implemented the Kubernetes stuff. I thought it was kind of overkill, but as the new guy I didn't feel it was my place to point it out. Buy hey, some people got some nice experience on their resumes.
Citus offered being able to have most of your database be plain postgres & then having the few tables that take up most the database be distributed
But for a small company you probably don't need scaling period
Also, are you sure your data is actually in a correct view across all these "realms"?
I’m admittedly using a strange architecture for my use case and I realize now commenting here opened too big a can of worms as explaining exactly what I’m doing would derail the thread. Suffice it to say, my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data.
But yes, using a more horizontal database strategy makes it very easy to manage data across realms. That’s one of the main benefits. A single DB would be much harder as far as isolation and separating test/production traffic (assuming this is what you mean by views) than having multiple separable dbs that only talk to dev/staging. And I can easily wipe out and isolate dev and staging this way. I’m frankly shocked people would advocate a single db that doesn’t allow you to do this.
That's chump change size even for a medium EC2/RDS instance, which should be capable of tens of millions of queries a day without the CPU or disk complaining at you (unless all your queries are table scans or unindexed).
> my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data
It doesn't matter, it's still just bytes. What will matter is your query pattern relative to the databases query planner efficacy, and how updates/deletes impact this.
> makes it very easy to manage data across realms
You can just as easily do this at first as separate databases/schemas on the same physical server, with different users and permissions to prevent cross-database/schema joins so that when you need to move them to different machines it's an easier process.
Everyone I know that has tested isolated multi-tenancy that wasn't dependent on legal needs ended up abandoning this approach and consolidating into as little hardware as possible. Heap Analytics had a blog post a few years ago about this, but I can't seem to find it.
Regardless, hope you success in your endeavor and that you come back in a few months to prove us all wrong.
If you have not launched yet but are optimizing for facebook-scale, that's not the optimal approach.
I can't comment on your database experience since I don't know it, but the vast, vast majority of people underestimate by orders of magnitude what a database can handle.
If you're not a large public company we all know about (and you're not, if you haven't launched yet), you don't need all the horizontal scale you seem to be building.
I remember joining one company (still a startup but a large one about to IPO). My day#1 briefing was about how this one database was in urgent need of replacement with a dozen+ node cassandra cluster because it was about to exceed it's capacity any second now. That was to be my highest priority project.
I took some measurements on usage and capacity and put that project on the backburner. The db was nowhere near capacity on the small machine it was running on. Company grew a lot, did an IPO, grew some more. Years later I left. That db was still handling everything with plenty of headroom left to grow more.
Something operates the database. That thing is going to need a database.
Postgres and etcd are very different. SQLite is very different. Look at this microk8s thread - https://github.com/canonical/microk8s/issues/3227 - as a canonical example. Context is they use dqlite, which is multi-master SQLite with raft. Here people come to actually use the data and it’s a disaster. Just use etcd!
Then people will deploy Postgres on top of Kubernetes. Great. Maybe they will deploy it differently, and their database is a bunch of files in /etc/. Whatever. But you can’t run Kubernetes on Postgres, you wouldn’t want to anyway, the choice to use etcd is very thoughtful. This is an environment designed to simplify the layers of bootstrapping, Kubernetes’s superpower is that it runs as much of its own stuff on top of itself as safe as it is to do so.
It still needs a database that is going to be separate from your applications, and unfortunately, for all interesting applications, the line between deployment and business-logical concerns is blurred.
So IMO, you will have multiple databases, because logical and deployment concerns are intertwined; deployment concerns need databases; and thus to solve chicken and egg, you’ll have at least 2, hopefully the 1st is super easy to administer.
Perhaps you disagree. Maybe you should use dqlite as the only database. It’s so naive to think anything is CP as long as there’s consensus in front of it. And if there’s no strong opinion about the arcane issue I just said, worse if it’s meaningless gobbledygook; or if the strong opinion is “use Postgres,” the data is screwed and the users are screwed.
So long as your application doesn't throw space and performance away at every opportunity, vertical scaling goes really far these days.
but you need some redundancy to avoid service downtime if that server/rack/datacenter will have a failure.
I see this a lot in RDBMSes. I've used plenty of 'em, and I have nothing against projects like PostgreSQL; they're amazing, especially for the price. But computers cost money, and maintenance time costs money, and development time costs money, and I know from experience that I can squeeze a lot more performance out of the same hardware using MSSQL than I can PostgreSQL, and I'll have an easier time doing it, too. I can even point to some of the specific design and feature differences that lead to the discrepancy.
It's to the point where, whenever I see an article talking about the limitations of relational databases, I automatically mentally insert a "that you can get for free" into the sentence.
Basically striping-mirroring RAID but for… data.
Undoubtedly the reality of widely available SQL systems today has not lived up to that original relational promise in the context of modern expectations for large-scale reactive applications - maybe (hopefully) that can change - but in the meantime it's good to see Rama here with a fresh take on what can be achieved with a modern 3GL approach.
That seems right to me, but is it really provably true?
Now you just need one tuple ("our_system", "the_db", the_binary_blob)
QED :)
Practically? I've never had someone throw a problem at me I couldn't model in SQL. Not saying I can guarantee performance, but in terms of abstract modeling I've never encountered something that can't be done "clean".
I'd like to use the analogy of painting. What can't you paint on that canvas that isn't trivially-hindered by the available media? Can you actually describe that space of things?
In my estimation, premature optimization is why most people can't work in these terms. You wind up with a tuple like WorldObjects and immediately go off the deep end about a billion row scale before you are done naming things.
To be truly universal, your model needs to be computationally stronger - it needs to be turing complete. For example fractals could never be modeled by any number of tuples, but can be perfectly represented by very short computer programs.
One hacky equivalence I can come up with is that you have a tuple with a single string value, and encode something in that string.
Obviously data structures vary, performance is required, and it becomes a bottleneck because it's so critical that mortals can't touch it and every change must be ultra-safe. And then there's security: what if someone finds a way from the development bug section to the HR or executive-only financials sections? Generally, anyone who has tried to implement universal ERM systems knows how difficult and painful integrated systems can be.
But those are the extremes. I'd be interested to know how far people have gotten, and how they did it, pursuing this ideal? I've never seen a business run on one. How about a personal knowledge-management system? Does everything fit? Do you still use spreadsheets for something 'quick', text files for free-form, etc.?
Amen. Previous co. had a cargo-cult microservices setup including a separate DB for each app. This made things unnecessarily complicated and expensive for no business benefit (definite resume-padding benefit). Lesson: Don't complicate things until you're forced to.
If you’re at that scale, then great, do that. But I also fully agree that cargo-culting that kind of decision out of the gate can be a massive waste of money.
If on the other hand, you're fine yolo bringing down your entire tech stack for potentially hours force upgrading all services at once, then you probably shouldn't be using microservices to begin with.
(And yes there are strategies to mitigate needing a big bang for renames as well as long as writers are all consistent, I just wanted to describe why everyone touching the same data stores leads to terrible outcomes like people being unable to make changes because "it's hard", which nobody says out loud but it's definitely visible in the tech stack).
A modern top tier RDBMS is an incredibly powerful, fast, mature, stable tool - when wielded wisely by the knowledgeable wizard. To be fair, the developers must have a solid understanding of programming language, and business domain, and SQL / data dictionary / ERD - which I've always taken for granted but eventually realized is not the norm.
Having a log store make sense, and having a super-VIEW replacement that can re-compute the data (so I can denormalize o build secondary tables/indexes that could or not be stored) is a common pattern.
That is fine.
What is NOT fine, and that normal DBS avoid, is to add "eventual consistency".
I NEED to get exact counts, sum, avg, and RIGTH NOW, not a second later.
The more "eventual" the data become the worse things are, and that is the major way to add bugs and problems.
One of the reasons FK are important is they eliminate the chance of getting the relationships wrong, but also, to get it right NOW.
There is the challenge of workload separation and scaling each component separately but that can be resolved by pulling out challenging workloads into their own "database" albeit on the same stack.
100% agreed. One of the biggest issues SQL databases have faced is that the scope & scale of "one transactional system" has evolved a lot more quickly than any off-the-shelf database architecture has been able to keep up with, resulting in an explosion of workaround technologies and approaches that really shouldn't need to exist.
We're now firmly in the cloud database era and can look at things like Spanner to judge how far away Postgres remains from satisfying modern availability & scaling expectations. But it will be great to see OSS close the gap (hopefully soon!).
No, it's all open source. YugabyteDB Anywhere is just automation & 24/7 core engineering support. All db features are open source. You can re-create Anywhere without needing to touch c/c++.
Note: All features have always been open source (besides incremental backup which will be soon).
Even if theoretically it was all you ever needed, the other constant problem is the implementation. Most developers, today anyway, are morons. They don't understand how databases work or how to use them, so they use them poorly. In order to get away from that fact, they invented new databases (like NoSQL) so they could use their brains less and just write more glue code. Turned out that was horrible as well.
Pretty soon the tides will turn yet again and "just having a big database" will again be seen as not in vogue and another paradigm of "simplicity" will come around (instead of "one big database" being simple it'll be "many small databases" like we had with microservices).
Those who don't understand history are doomed to repeat it.
In contrast, pocketbase [0] was discussed two days ago, and there are many user-friendly alternatives for relational data [e.g. 1-4]
[0] https://news.ycombinator.com/item?id=38898934
[1] https://www.getgrist.com/
[2] https://nocodb.com/
[3] https://baserow.io/
[4] https://www.rowy.io/
I don't even think that that proves that this cannot be done... And I've seen A LOT of people try. But, things invariably start to break down, and at this point I just think that the statement above is more wishy washy than anything, though I want to believe...
Please no buzzwords like "paradigm shift" or "platform". If diagrams are necessary, I'd love to read a post that explains clearer.
(Though NoSQL has outlived its usefulness as a concept IMO, it's just too loose to be useful beyond its early use for "something like CouchDB/Mongo", which this is clearly not)
You have a "Depot", which is an append-only log of events, and then build arbitrary views on top of it, which they call "P-States". The Rama software promises low-latency updates of these views. Applications built on this would query the views, and submit new events/commands to the Depot.
Rama runs as a cluster, and any number of applications (called "modules") are deployed onto that cluster. Deep and detailed telemetry is also built-in.
The programming model of Rama is event sourcing plus materialized views. When building a Rama application, you materialize as many indexes as you need as whatever shapes you need (different combinations of durable data structures). Indexes are materialized using a distributed dataflow API.
Since Rama is so different than anything that's existed before, that's about as good of a high-level explanation as I can do. The best resource for learning the basics is rama-demo-gallery, which contains short, end-to-end, thoroughly commented examples of applying Rama towards very different use cases (all completely scalable and fault-tolerant): https://github.com/redplanetlabs/rama-demo-gallery
Is this basically an RBDMS and Kafka in one? Can I use SQL?
I understand the handwaving around programming semantics, but I'd like clearer explanations of what it actually is and how it works. Is this a big old Java app? Do you have ACID transactions? How do you handle fault tolerance?
It may be early, but I believe folks will be curious about benchmarks. And maybe, someday, Jepsen testing.
- Public build that you can download and run yourself locally https://redplanetlabs.com/docs/~/downloads-maven-local-dev.h... - rama-demo-gallery, containing short, thoroughly commented examples in both Java and Clojure https://github.com/redplanetlabs/rama-demo-gallery - Gentle six-part tutorial introducing the concepts and API, including how to run stuff locally https://redplanetlabs.com/docs/~/tutorial1.html - Introduction to the first-class Clojure API https://blog.redplanetlabs.com/2023/10/11/introducing-ramas-...
Here are a few pages related to fault-tolerance:
- https://redplanetlabs.com/docs/~/replication.html - https://redplanetlabs.com/docs/~/microbatch.html#_operation_... - https://redplanetlabs.com/docs/~/stream.html#_fault_toleranc...
Especially since you chose to use the name Rama, I am wondering whether this will be for the benefit of all, or only for the benefit of the few who already control more than a fair share of power(finances)?
That's… a database…
I mean, seriously, how is this not a database?
Just with a lot of modern technology around scaling, logging etc. which hopefully (I haven't used it yet) eliminates all the (many many) issues this approach had in the past.
not quite, through more like anything which is widely known
I have worked (a small bit) on very similar (proprietary non public internal) systems before ~5 years ago and when doing so have read block-posts about the experience some people had with similar (also proprietary internal) systems which at that point where multiple years old ...
I guess what is new is that it's something you can "just use" ;=)
But yeah, quite a lot of hype and red flags. My favorite from the website: "Rama is programmed entirely with a Java API – no custom languages or DSLs." And when you look at the example BankTransferModule.java: > .ifTrue("isSuccess", Block.localTransform("$$funds", Path.key("toUserId").nullToVal(0).term(Ops.PLUS, "*amt")))
Yeah, it's probably fair to call that a DSL, even if entirly java.
Anyway, hope to get the chance to work with event based systems one day and who knows, maybe it will be Rama.
However, there is just more going on in an event sourcing model. Instead of saving data to a location and retrieving it from that location you save data to one location, read it from another location, and you need to implement some sort of linker between the two (or more).
This also comes down to my personal subjective experience. I actually really like event sourcing but I have worked on teams with these systems and I have found that the majority of people find them much harder to reason about than traditional databases.
(declare-depot setup *my-events (hash-by :user-id))
That's it, and you can make as many of those as you want. And here's how a topology (a streaming computation that materializes indexes based on depots) subscribes to that depot:
(source> my-events :> *data)
If you want to subscribe to more depots in the topology, then it's just another source> call.
That these are integrated and colocated also means the performance is excellent.
Seeing Closure ironically makes me think of Twitter though.
Right, so the solution is more complexity? Of course it is. Sigh
More complexity? The writer makes it very simple. Just use their product Rama.
Our Twitter-scale Mastodon implementation is a direct demonstration of this. It's literally 100x less code than Twitter had to write to build the equivalent feature-set at scale, and it's more than 40% less code than Mastodon's official implementation. This isn't because of being able to design things better with the same tooling the second time around – it's because it's built using fundamentally better abstractions.
This is also true of most of the main SQL database engines, no?
I fundamentally disagree with the premise of your blog post but not the premise of your company, so why don't I make an ask for you to write a much more practically useful application. Design a basic shopping cart using your system and compare and contrast it with a well designed relational equivalent. A system that allows products to be purchased and fulfilled is a far closer match to what the majority of companies are using software for compared to writing a Twitter clone.
Here's my take -- at a certain point in scale and volume using a database, it actually does make sense to rewrite all of the following from scratch:
- query planning
- indexing (btree vs GIN, etc) and primary/foreign/unique key structure
- persistence layer
- locking
- enforcing ACID constraints
- schema migrations/DDL
- security, accounts and permissioning
- encryption primitives
But more crucially, my belief and experience is that most companies making lots of money from software products will NEVER remotely reach that scale -- and prematurely optimizing for that is not just the wrong decision but borderline professional malpractice. You can get very far with Postgres and JSONB if you really need it, and you'll spend more time focusing on your business logic than reinventing the wheel.
I'd like to be proven wrong. But I get sinking feeling that I'm not wrong, and while your product is potentially valuable for a very specific use case, you're doing your own company a disservice by distorting reality so strongly to both yourselves and your prospective customers.
I'll round out this comment by linking another comment in this thread that goes very well into the perils of event sourcing when the juice isn't worth the squeeze:
https://news.ycombinator.com/item?id=38930591
We started with a Twitter demonstration because: a) its implementation at scale is extremely difficult, b) I used to work there and am intimately familiar with what they went through on the technical end, and c) the product is composed of tons of use cases which work completely differently from each other – social graph, timelines, personalized follow suggestions, trends, search, etc. A single platform able to implement such diverse use cases with optimal performance, at scale, and in a comparatively tiny amount of code is simply unprecedented.
It's a very impressive demo. You should be proud, keep your chin up, and keep us updated as Rama's value prop. continues to grow.
You're not really addressing the substance of my comment so I am left to assume that your omission is because you cannot address it. Be that as it may, you'll hopefully at least take my final comment at its face value that you do your own product a disservice by overhyping what its use case is towards areas that it is objectively a poor fit. People have tried event sourcing many times and it's just not a good fit for many if not most workloads. You can't be everything to everyone. There's nothing wrong with that.
My advice to you is this: call out the elephant in the room and admit that, and focus on workloads where it is a good fit. That extra honesty will go a long way in helping you build a business with sustainable differentiation and product market fit.
This "Twitter-scale mastadon implementation" is when my red flags went up. It's meant to demonstrate a simpler and more performant architecture, but it actually demonstrates "things you should never do" #1: rewrite the code from scratch.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
"The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed."
The "1M lines of code" and "~200 person-years" of Twitter being trashed on in this article is the outcome of Twitter doing the most important thing that software should do: deliver value to people. Millions of people (real people, not 100M bots) suffered thru YEARS of the fail-whale because Twitter's software gave them value.
This software has only delivered some artificial numbers in a completely made-up false comparison. Okay it's built on "fundamentally better abstractions", but until it's running for people in the real world, that's all it is: abstract.
Please don't tout this as a demonstration of how to re-create all of Twitter with simpler and more performant back-end architecture.
- We stress-tested the hell out of our implementation well beyond Twitter-scale, including while inducing chaos (e.g. random worker kills, network partitions). - We ran it for real when we launched with 100M bots posting 3,500 times / second at 403 average fanout. It worked flawlessly with a very snappy UX.
The second-system effect is a real thing, but there's a difference when you're building on radically better tooling. All the complexity that Twitter had to deal with that led to so much code (e.g. needing to make multiple specialized datastores from scratch) just didn't exist in our implementation.
That is exactly correct. All testing is make believe except for real case studies by real customers with an intent to pay, and barring that, real pilots with real utilization. Otherwise, you run the risk of building a product for a version of yourself that you are pretending is other people.
At some point in time the same argument was made for relational databases despite there being stable systems built without them based on ISAM. The newer relational systems took a lot less work to implement but that didn't imply that it made sense to rewrite the ISAM based systems.
Joel was talking about commercial desktop software in the extremely competitive landscape of the 90s, he wasn't talking about world-scale internet service infrastructure. The architecture that delivered a set of X features to the initial N users isn't always going to be enough for X+Y features to the eventual 1000*N users that you promised your investors.
Companies like Google are quite public about how much they rewrite internal software, and that's just what the public hears about. A particular service might have been written with all of the care and optimization that a world-class principal engineer can manage, serve perfectly for a particular workload for several years, and yet still need to be entirely replaced to keep up with a new workload for the following several years.
You wouldn't tell another engineer that they shouldn't rewrite a single function because software should never be rewritten, so it also doesn't make sense to tell them not to rewrite an entire project either. It's their call based on the requirements and constraints that they know much more about.
Nobody should be rushing out to rewrite Linux or LLVM from scratch, and yet we wouldn't even have Linux or LLVM if their developers didn't find reasons to create them even while other projects existed. In hindsight it's clear those projects needed to be created, but at the time people would have said you should never rewrite a kernel or compiler suite.
If you can convince more developers to apply what is effectively canonical store -> workers -> partitioned denormalized materialized views as a pattern where it makes sense, then great.
But you can do that with just the tools people already have available. Heck, you can do that with just multiple postgres servers (as the depot, and for the "p-stores" and for the indexing functions), and then you don't need to ditch the languages people are familiar with for both specifying the materialization and the queries.
Part of the reason we use a "hodgepodge of narrow tooling" however, tends to be that it allows us to pick and choose languages, and APIs, depending on what developers are familiar with and what suits us, and it allows people to pick and mix. Convincing people to give up that in favour of a fairly arcane-looking API restricted to the JVM is going to be a tough sell a lot of places.
Does your implementation match the features of Mastodon?
Yes, we implemented the entirety of Mastodon from scratch.
But, at a certain level of scale everything is a data engineering problem, and sometimes this is the (relatively) simple solution when viewed in the context of the entire system.
'Just use mySQL/SQLite/Postgres' is great advice until it isn't.
I think it would a mistake to approach a domain and assume nothing can be made simpler or more straightforward.
I genuinely believe we would all be better off if Martin Fowler's original article about Event Sourcing had never been written. IMO, it's a bad idea in 99% of cases.
One thing there is: These were massive engineering departments in very large companies. Think of several dozen data processing teams, each very much their own small company, all working on different domains with event and stream consumers and utilizing the possibility of restarting their stream consumers some time ago. Impressive Kafka-Clusters ingesting thousands and thousands of events per second and keeping quite the backlog around to enable this refeeding, outages and catching back up and such. At their scale and complexity, I can see benefits.
However, at that scale, Kafka is your database. And you end up with a different color of the same maintenance work you'd have with your central database. Data ends up not being written, data ends up being written incorrectly, incorrectly written data now ends up with transitive errors. At times, those guys ended up having to drop a significant timespan of data, filter our the true input events, pull in the true events and start refeeding from there - except then there was a thundering herd effect, and then they had to start slowly bringing up load... great fun for the management teams of the persistence.
Note that I'm not necessarily saying e.g. Postgres is the solution to everything. However, a decently tuned and sized, competently run Postgres cluster allows you to delay larger architectural decisions for a long time with run-of-the-mill libraries and setup. Forever in a nonzero amount of projects.
We use SQLite and Postgress for pretty much everything. And about 90 percent of the server code is auto generated using an inhouse code generator.
Why does anyone create a blog like that?
From what I can tell in the article it seems their differentiator is Event Sourcing and having arbitrary complex index builders on top of the events. It seems similar to EventStoreDB[1].
I have always been interested by the concept of an event sourcing database with projections and I want to build one eventually so it is interesting to see how they have approached the problem.
Also they mention on their site:
> Rama is programmed entirely with a Java API – no custom languages or DSLs.
It makes sense why they have gone this route if they want a "Turing-complete dataflow API" but this can be a major barrier to adoption. This is a big challenge with implementing these databases in my opinion because you want to allow for any logic to build out your indexes/projections/views but then you are stuck between a new complicated DSL or using a particular language.
1: https://developers.eventstore.com/server/v23.10/#getting-sta...
Also, Rama has a first-class Clojure API (I should probably update the website).
https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/rama-demo-gallery
Why take that path?
I get your point about global mutable state, but I do not see how you have done anything more than shuffle the complexity around
Being dependant on software like the JVM makes me very reluctant to investigate further. The JVM is not involved in anything I use, I do not think i want it.
That was my reaction as well. The article's claimed argument against databases is that they are global mutable state, which is supposed to be bad. But none of what the article advocates for gets rid of global mutable state (which of course you can't do in any real world application, because the real world itself is global mutable state and anything that deals with the real world has to deal with that). It's just hawking a particular way of storing global mutable state.
I.e. I did like how Clojure had it's atomically updating refs.
And you realisticaly won't get more principled than with traditional RDBS, with it's ability to transactions, rollbacks, e.t.c.
Only alternative that I have seen proposed to using DBAS while not relaxing the guaranties, is to double down on event-sourcing and not having mutable-state anymore. Everything could be just a fold over all of the incomming events, right? But I don't think I have seen non-toy example of that anywhere I worked at.
Event sourcing doesn't get rid of mutable state. The event store gets mutated every time a new event comes in.
As far as the term "global mutable state" is concerned, which is what the article claims is bad, the event store is global mutable state. One can think of the individual event records in the store as immutable (they don't change once they're recorded), but the article is talking about the entire store, not an individual record in it.
> your copy of the data at point X will always be valid
If your copy of the event store is out of date (missing some events), it's not valid. A new event that your copy hasn't yet registered might change the results of queries on the store, meaning that your copy will be delivering out of date, hence invalid, query results. You can't handwave around that by gerrymandering the definition of "immutable". The simple fact is, as I said, that the real world is global mutable state, and if your app is dealing with the real world, it has to deal with that.
And conceptually, you can then push all of those "what if I am missing events, or tey are in wrong order" into the fold over the data.
Because real world is mutable data, but history of the world at any particular point isn't. It can be incomplete, it can be a personal view, I like that there are things made explicit that otherwise wouldn't. Nice abstraction.
Like, if you are splitting hairs, is even array-append in Haskell immutable? It does create new data somewhere in the memory. But I don't think that makes Okasaki's thesis and book any less valuable :)
Close enough to what? I get that there are advantages to storing global mutable state this way. I just don't see the point of denying that what you are doing is storing global mutable state--every time you append a new log entry to the data store, you mutate the state.
> history of the world at any particular point isn't
Sure, I get that too, and I get that it's nice to have that stored explicitly instead of having to reconstruct it for the particular cases where you need it.
But if someone is running a query on your database, except in some particular cases, they don't want the history of the world at some particular point to dictate the response. They want the most up to date history of the world to dictate the response. And that changes every time you append a new log entry to the data store. Again, I don't see the point of denying that that's what is going on.
> is even array-append in Haskell immutable?
The point of "immutable" in Haskell is that every object is immutable. So array-append in Haskell does not take an existing array object and append an element to it (as the corresponding operation would for, say, a list in Python). It constructs a new array object that is the same as the old object with the new element appended. So the old array object is not mutated.
But the array-append operation of course does change the global state of the program, as you say. So there is no such thing as an immutable global state of a Haskell program. And of course the same would be true for a program in any language.
> I don't think that makes Okasaki's thesis and book any less valuable
Of course it doesn't. But it also doesn't mean the definition of "immutable" that he is using is the only possible or useful one. The article under discussion here is using a different definition, which for the topic it is discussing is more useful. Once more, I don't see the point of trying to twist words around to deny that.
I reread the article twice now, and I don't see a conflicting definition, that would contradict i.e. append-only-log being immutable.
Unless your argument is that the author is in-denial about benefits of immutability, because it can't be achieved in this weirdly strict form you seem to propose?
But maybe I need to reread second paragraph of the article one more time?
Yes, I know, and it may well be a good solution to various issues with conventional databases. I just don't see how it's a solution to global mutable state, since there still is global mutable state.
> I don't see a conflicting definition
The article never gives an explicit definition of "global mutable state". It also does not explicitly say that its proposed solution gets rid of global mutable state. The section "Global mutable state is harmful" strongly implies that its proposed solution will somehow get rid of global mutable state, but the article never actually says that it does. It just goes into a bunch of other benefits of event sourcing and materialized views, all of which might well be valid benefits, but none of which amounts to getting rid of global mutable state.
> Unless your argument is that the author is in-denial about benefits of immutability
I can't see into the author's mind, so I don't know if the aspects of the article that I described above were inadvertent, because the author simply hasn't really thought through what "global mutable state" actually means, or whether it's on purpose, to imply a claim about the article's proposed solution that sounds good but isn't actually true, as a marketing tactic.
As I said in my earlier response, the article doesn't explicitly define what "global mutable state" is, but it does say that conventional databases are global mutable state the same way that global variables in an ordinary program are.
By that definition, an append-only log as a "single point of truth" data store, which is the article's proposed "solution", is also global mutable state. The article does not acknowledge this, but it seems obvious to me.
Yes, there is. Appending a new log entry to the depot mutates the depot. It doesn't mutate earlier records in the depot, but the depot is the entire record set, not just individual records. The record set is different after the append than before it.
You can also do C++ and Rust, but that does require a bit of binding work.
So it's not really limiting you to a single language, or these days, even a single runtime.
(A replicated set of databases for the depots; a set of workers creating the "p-stores" in ... another set of databases, and you've replicated the architecture; heck you can even run this entirely off postgres w/replication + triggers to handle partitioning and materialization with no external workers if you like)
EDIT: Also their API fairly unambiguously is effectively a DSL, just one without it's own separate syntax and parser allowing access from outside the JVM.
Really? I never had problems using some JVM base image and deploying via a Docker image, which is what I would be doing anyway.
> If it provided some massive major advantage over the large number of other options, I'd consider it
People reject using technologies because of the impact of having to support them all the time when the benefit does not justify needing to support one more technology. I don't have an issue with using JVM based services if I can have someone else manage them for me, but when I have to deal with the operational issues myself, or within my team, then how comfortable we are with managing a specific technology vs. the advantages it brings over competing alternatives absolutely is part of the consideration, and it'd be irresponsible for it not to be.
In this case the benefit seems marginal at best, and I'm not deploying a JVM based service for marginal benefits given how much hassle they've brought me over the years.
Nothing is outright rejected because it runs on the JVM, but it is a detracting quality, adding complexity. If Kafka is clearly the best solution available for your problem, you'll suck it up and the use JVM, but if there is another approach that will also serve your needs without a JVM dependence then the choice is easy.
However, the jump from that to saying you have it is a bit like the jump from talking about the messiah and saying that he's sitting in the next room just now.
We investigated event sourcing and materialized views as the backbone of a (new) big business application five years ago. The problem was that there were so few products back then and they were so untested (unless my mind is playing tricks on me, we did look at EventStoreDB, as well as Kafka streams), and despite being open source they were also usually heavily tied to one commercial provider. No one had anything like experience with it. I couldn't convince my colleagues to go for this, much less my bosses. Hell, I could barely convince myself.
Philosophically, I believe it's sound, but how much has changed in those years?
I stopped reading the article once it was evident that the author was not making any case regarding databases being unnecessary, and instead was putting up a series of strawmen that sounded like desperate attempts to reject all alternatives to the thing they were selling.
The author's attempt to talk about data structures was particularly baffling, knowing that pretty standard RDBMS such as PostgreSQL already support JSON data types. Are we supposed to forget about that?
That's kind of the point. Model your data. Think about it. Don't (mis)treat your database as a "persistence layer" -- it's not. It's a knowledge base. The "restriction" in the relational model is making you think about knowledge, facts, data, and then structure them in a way that is then more universal and less restrictive for the future.
Relations are very expressive and done right is far more flexible than the others named there. That was Codd's entire point:
https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf
"Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation) ..." and then goes on to explain how the predicate-logic based relational data model is a more universal and flexible model that protects users/developers from the static impositions of tree-structured/network structure models.
All the other stuff in this article is getting stuck in the technical minutiae of how SQL RDBMSs are implemented (author seems obsessed with indexes). But that's somewhat beside the point. A purely relational database that jettisons SQL doesn't have to have the limitations the author is poking at.
It's so frustrating we're still going over this stuff decades later. This was a painful read. People developing databases should already be schooled in this stuff.
Love this. Your database is your app in my opinion. The data model matters.
RDBMS hasn’t stuck around this long only because it’s good enough. It’s an incredibly powerful way to model data, for now and the future, but yes, it does require you to think carefully and slow down. These are not things to be avoided.
And so people go looking for something with a modern syntax, that's properly composable, functional, etc. but because they haven't studied the foundations, they reinvent the wheel badly.
The Codd paper is so readable, and so persuasive. People need to start there, and then argue why their novel approach is better on first principles, not just because it smells better than SQL.
But that it's stuck around and done so well for so long tells you that the core foundation is sound.
The more I read it the less it makes sense.
> A purely relational database that jettisons SQL doesn't have to have the limitations the author is poking at.
Agreed. Relational databases can take us a lot further yet.
Really, they're not avoiding modelling the data. They're modelling it in their "P-stores" whether they want to admit that's just another data model or not. It's clearly not data models they object to, but typical database DDL coupled with a desired to not throw away data that doesn't conform right now. Depending on your application domain, not throwing away data that doesn't conform to your current data model can be good or bad, but it's not an accident that they picked Twitter/Mastodon where it's easy to justify not rejecting non-conforming documents.
I agree with you that this doesn't require ditching relational models, though. For that matter not even SQL. You can even fairly easily build this kind of architecture without e.g. ever leaving Postgres (replicated pair for their "depots"; triggers + postgres foreign data wrappers to materialize views on index servers and/or to do requests across index servers).
Nothing in Rama prevents you from doing that. You just see it through the SQL lens and your entire point is invalid.
I just happen know what a "relation" is, while the author doesn't seem to.
If so I'd be curious how they've solved making it in anyway operational less complex to manage then a database. It's been a few years since I've run Kafka but it used to kind of be a beast.
[1] https://redplanetlabs.com/docs/~/why-use-rama.html#gsc.tab=0
The first is our Twitter-scale Mastodon implementation, which is 100x less code than Twitter wrote to build the equivalent at scale (just the consumer product). It's also more than 40% less code than Mastodon's official implementation (which isn't scalable). https://github.com/redplanetlabs/twitter-scale-mastodon
The rama-demo-gallery repo also contains many short, self-contained, thoroughly commented examples of applying Rama towards very different use cases. These include user profile management, time-series analytics, atomic and fault-tolerant bank transfers, and more. https://github.com/redplanetlabs/rama-demo-gallery
"Global mutable state is harmful" - well... yes, that's totally correct. "The better approach [..] is event sourcing plus materialized views." .....errr... that's one approach. we probably shouldn't hitch all our ponies to one post.
"Data models are restrictive" - well, yes, but that's not necessarily a bad thing, it's just "a thing". "If you can specify your indexes in terms of the simpler primitive of data structures, then your datastore can express any data model. Additionally, it can express infinite more by composing data structures in different ways" - perhaps the reader can see where this is a bad idea? by allowing infinite data structures, we now have infinite complexity. great. so rather than 4 restrictive data models, we'll have 10,000.
"There’s a fundamental tension between being a source of truth versus being an indexed store that answers queries quickly. The traditional RDBMS architecture conflates these two concepts into the same datastore." - well, the problem with looking at it this way is, there is no truth. if you give any system enough time to operate, grow and change, eventually the information that was "the truth" eventually receives information back from something that was "indexing" the truth. "truth" is relative. "The solution is to treat these two concepts separately. One subsystem should be used for representing the source of truth, and another should be used for materializing any number of indexed stores off of that source of truth." this will fail eventually when your source of truth isn't as truthy as you'd like it to be.
"The restrictiveness of database schemas forces you to twist your application to fit the database in undesirable ways." - it's a tool. it's not going to do everything you want, exactly as you want. the tradeoff is that it does one thing really specifically and well.
"The a la carte model exists because the software industry has operated without a cohesive model for constructing end-to-end application backends." - but right there you're conceding that there has to be a "backend" and "frontend" to software design. your models are restrictive because your paradigms are. "When you use tooling that is built under a truly cohesive model, the complexities of the a la carte model melt away, the opportunity for abstraction, automation, and reuse skyrockets, and the cost of software development drastically decreases." - but actually it's the opposite: a "cohesive model" just means "really opinionated". a-la-carte is actually a significant improvement over cohesion when it is simple and loosely-coupled. there will always be necessary complexity, but it can be managed easier when individual components maintain their own cohesion, and outside of those components, maintain an extremely simple, easy interface. that is what makes for more composable systems that are easier to think about, not cohesion between all of the components!
"A cohesive model for building application backends" - some really good thoughts in the article, but ultimately "cohesion" between system components is not going to win out over individual components that maintain their cohesion and join via loosely-coupled interfaces. if you don't believe me, look at the whole Internet.
In theory ES is brilliant and offers a lot of great functionality like replaying history to find bugs, going back to any arbitrary point in history, being able to restore just from the event log, diverse and use case tailored projections, scalability, ...
In practice it increases the complexity to the point were it's a pointless chore.
Problems:
* the need for events, aggregates and projections increases the boilerplate tremendously. You end up with lots of types and related code representing the same thing. Adding a single field can lead to a 200+ LOC diff
* a simple thing like having a unique index becomes a complex architectural decision and problem ... do you have an in-memory aggregate? That doesn't scale. Do you use a projection with an external database? well, how do you keep that change ACID? etc
* you need to keep support for old event versions forever, and either need code to cast older event versions into newer ones, or have a event log rewrite flow that removes old events before you can remove them from code
* if you have bugs in you can end up needing fixup events / event types that only exist to clean up , and as above, you have to keep that around for a long time
* similarly, bugs in projection code can mess up the target databases and require cumbersome cleanup / rebuilding the whole projection
* regulation like GDPR requires deleting user data, but often you can't / don't want to just delete everything, so you need an anonimizing rewrite flow. it can also become quite hard to figure out where the data actually is
* the majority of use cases will make little to no use of the actual benefits
A lot of the above could be fixed with proper tooling. A powerful ES database that handles event schemas, schema migrations, projections, indexes, etc, maybe with a declarative system that also allows providing custom code where necessary.
I'll take a look at Rama I guess.
By "exactly-once processing semantics", I mean that regardless of how many failures are on the cluster (e.g. nodes losing power, network partitions), the updates into all partitions of all indexes will be as if there were no failures at all. Pure streaming, on the other hand, has either "at-least once" or "at-most once" semantics.
The schema changes are hard (e.g. try to normalize/denormalize data), production is the only environment when things go wrong, in-place changes with untested revert options are default, etc.
It's possible that in turn forces users to adapt and split their changes into smaller chunks, but that's pretty speculative and they might simply fail entirely in the attempt.
Which data stores are more easily evolved than a relational DB with a fairly restrictive schema?
I just wish there was a better tooling to evolve schema. Blue/green deployments, schema as code (no manual up and down procedures), good backward compatibility story, shadow testing in production, etc.
The database still feels like a monolith and did not get the benefits of the microservice revolution.
https://www.datomic.com/ https://xtdb.com/
I imagine that will bring great tax benefits to programing schools.
There is no option to just "put it all in a database". You need to compose a number of different systems. You use your individual databases as indexes, not as primary storage, and the primary storage is probably S3. The post is interesting and the author has been working on this stuff for a while. He wrote Apache Storm and used to promote some of these concepts as the "Lambda architecture" though I haven't seen that term in a while.
* The immutability, lambda architecture points I agree with. I think the separation of the immutable log from the views is important. Databases are frequently used in ways that go against these principles.
* I am not sold that being unable to express the domain model correctly is really a fair criticism of databases. Most businesses in my experience have a domain that is modeled pretty well in a relational DB. I haven't seen a better general solution yet, though I haven't checked out Rama.
At the low end of the scale, there are a lot of companies (or projects) for which the entire dataset fits in a single managed Postgres instance, without any DBA or scalability needs. They still suffer from complexity due to mutable state, but the architectural separation of source of truth vs "views" can be implemented inside the one database, using an append only table and materialized views. There are some kinds of data that are poorly modeled this way (e.g images) but many kinds that work well.
So I don't really view the architectural ideas as repudiating databases in general, more as repudiating a mutable approach to data management.
> You use your individual databases as indexes, not as primary storage, and the primary storage is probably S3.
Which is a perfectly valid use for a database. Our company's document management system uses a big database for metadata and then, of course, stores the actual files on disk.
It’s also probably fine for about 95% of companies, and that figure is rising.
Does the indexes have read after write guarantees?
I can't say I understand the example. It seems to talk a lot about implementation details compared to the naive SQL:
Who hurt you?
Databases have materialized views though, that solves this problem.
And yes, Rama's queries are a massively superior approach. The need for complex query planners is a result of limitations in how you index data, oftentimes from the tension between normalization and denormalization. With Rama, it's easy to robustly materialize multiple views that are already in the shape needed for queries.
Check out our tutorial for a gentle introduction to Rama's concepts https://redplanetlabs.com/docs/~/tutorial1.html
And I'm sorry, you won't convince me something defined in Java is superior to declarative SQL. There's a lot of problems with SQL, no doubt about it.
Any time I would save with this is wasted on learning java and this framework.
There isn't anything wrong with databases.
To claim that something static is "Twitter-scale" is just misleading IMO.
> To demonstrate its scale, we operated the instance with 100M bots posting 3,500 times per second at 403 average fanout.
The linked post about their demo [1] has more details about the challenges involved, including rendering home timelines.
[1]: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...
To me, this looks like an example of those ads that look like this
edit ( tried to do a text table right on here but it got really messed up in the page display)
here is a screenshot of the effort
https://imgur.com/a/XtwSkyx
> It’s common to instead use adapter libraries that map a domain representation to a database representation, such as ORMs. However, such an abstraction frequently leaks and causes issues. ...
FWIW, I'm creating a tool (strategy) that is neither an ORM or an abstraction layer (eg JOOQ) or template-based (eg myBatis). Just type safe adapters for normal SQL statements.
Will be announcing an alpha release "Any Week Now".
If anyone has an idea for how to monetize yet another database client library, I'm all ears. I just need to eat, pay rent, and buy dog kibble.
FWIW jOOQ's model worked great from a consumer/end-user point of view. I got to learn the whole library with no real strings attached, and then pay to adapt it/support it on Big Fucking Databases(TM) as projects required it. Their site and documentation is also quite pleasant.
In a way it feels like double-piggybacking: Microsoft, Oracle, et al. spend a lot on mindshare to make sure technical leads choose those respective stacks. Then the army of developers/consultants, required to get the promised ROI out of those stacks, inevitably have to go looking for tools. Be there to sell them the shovels. It helps if they are already familiar with it. (Free OSS adapters / "community" editions, etc., or even just very generous "evaluation" versions with no real effort to police use in a lab environment.)
Not sure how successful jOOQ has been financially, but considering they've been around for many years at this point, I have to imagine it's worked out well enough to pay for the lights and kibble?
The only other "dev tool" company I've enjoyed working with _more_ has been JetBrains. (The recent AI plugin not-withstanding.)
The domain knowledge embodied in JOOQ is kind of daunting. Truly huge.
Of the current SQL client API tools, mine is most like sqlc. Mine's major improvement, IMHO of course; but concept and workflow are comparable.
What do you think of paying to use such a tool for builds? eg Deployed to a CI/CD pipeline.
Free to use for sql-fiddle, personal, desktop, FOSS, etc. But once a project starts to use a tool for "real work", that's proof of value. Right?
Part of my just wants to want to do shareware. I published shareware in the early 1990s. Since I have the biz acumen of a sea cucumber, I just did the work and let people decide. It was great gig while it lasted.
I guess today's equiv would be some kind of patreon. My reluctance there is the PR/promotion/social media part. I have trouble even imagining what that'd look like.
It pays roughly EUR 400k ARR
I mostly like the approach, but there are a lot of questions/issues that spring to mind (not that some of them don't already have answers, but I didn't read everything). I'll list some of them:
* I'm pretty sure restrictive schemas are a feature not a bug, but I suppose you can add your own in your ETL "microbatch streaming" implementation (if I'm reading this right, this is where you transform the events/data that have been recorded to the indexed form your app wants to query). So you could, e.g., filter out any data with invalid schema, and/or record and error about the invalid data, etc. A pain, though, for it to be a separate thing to implement.
* I'm not that excited to have my data source and objects/entities be Java.
* The Rama business model and sustainability story seem like big question marks that would have to have strong, long-lasting answers/guarantees before anyone should invest too much in this. This is pretty different and sits at a fundamental level of abstraction. If you built on this for years (or decades) and then something happened you could be in serious trouble.
* Hosting/deployment/resources-needed is unclear (to me, anyway)
* Quibble on "Data models are restrictive": common databases are pretty flexible these days, supporting different models well.
* I'm thinking a lot of apps won't get too much value from keeping their events around forever, so that becomes a kind of anchor around the neck, a cost that apps using Rama have to pay whether they really want it or not. I have questions about how that scales over time. E.g., say my has depot has 20B events and I want to add an index to a p-state or a new value to an enum... do I need to ETL 20 billion events to do routine changes/additions? And obviously schema changes get a lot more complicated than that. I get that you could have granular pstates but then I start worrying about the distributed nature of this. I guess you would generally do migrations by creating new pstates with the new structure, take as much time as you need to populate them, then cut over as gradually as you need, and then retire the old pstates on whatever timeline you want.... But that's a lot of work you want to avoid doing routinely, I'd think.
I'm starting to think of more things, but I better stop (my build finished long ago!)
* Rama's JVM-based, so the entire ecosystem is available to you. You can represent data as primitive types, Java objects, Protobuf, Clojure records, etc.
* You deploy and manage your own Rama clusters. The number of nodes / instance types depends on the app, but Rama doesn't use more resources than traditional architectures combining multiple tools.
* Some databases support multiple very specific data models (e.g. Redis). I don't consider that flexible compared to Rama, which allows for arbitrary combinations of arbitrarily sized data structures of arbitrary partitioning.
* Depots (the "event sourcing" part of Rama) can be optionally trimmed. So you can configure it to only keep the last 2M entries per partition, for example. Some applications need this, while others don't.
* If you're adding a new PState, it's up to you how far back in the depot to start. For example, you could say "start from events appended after a specific timestamp" or "start from 10M records ago on each partition".
* We have a first-class PState migrations feature coming very soon. These migrations are lazy, so there's no downime. Basically you can specify a migration function at any level of your PStates, and the migration functions are run on read. In the background, it migrates iterates over the PState to migrate every value on disk (throttled so as not to use too many resources).
https://redplanetlabs.com/docs/~/tutorial1.html#gsc.tab=0
This is quite complex compared to setting up Postgres or mysql and sending some sql over some port.
I’m not sure I get what they are selling.