Am I just dense, or was the idea the CEO had just not a good idea and simply garden variety premature optimization?
Also, it seems unfair to say that the database ruins all good ideas when there continues to be significant impressive innovation in the space in cloud services. Like Amazon Aurora for one. If you really want to treat your RDBMS like an elastically scalable (with significant caveats) resource priced by the request, you can.
CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.
Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument when it comes to any production system that it isn't even worth discussing.
The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem.
Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).
GC pauses are not the only GC issue. Thrashing the caches and blowing up memory usage by an order of magnitude can be also very bad for performance. In a database system memory is very precious - the more of it you can use for caching / buffering users data, the better the performance.
As for the subsecond spikes in latency, these tend to multiply in a distributed system. If serving a client request takes N internal requests, the likelihood of hitting a GC pause somewhere is much larger than if you did only one local request.
Not sure about Go, but none of the "free" Java's GC guarantees low pauses. There is STW fallback even in the most advanced ones like ZGC. So you never know when it stops for more than 1s.
Is it a hard guarantee or just a soft goal like in G1, which tries to stay within the target pause milliseconds, but there are many ways it can fail to? Can it handle 100+ GB large heaps?
Even if it is a hard guarantee, then, from the link you posted, it is not even generational, so it will scan the whole heap quite frequently, and that is going to influence average performance quite visibly - you definitely dont want a database system to access all its cached memory once in a while.
Database systems are really all about memory and I/O management. You shouldn't outsource those core features to a universal algorithm, unless you wish to forgo any competitive advantage (at least in performance department). So this pushes the devs into the off-heap manual memory management territory, where dragons live (at least in Java, again - maybe Go is better in that regard). I've been there, and I don't recommend.
> Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument
False. All popular databases are written in C, and continue because of the GC issue. I would not use a general-purpose database written in Java because of GC, for example. We'll see how well Go works in practise.
> sub second spikes in latency
Go is supposed to be sub-second GC pause latency, but understand that most SQL queries are sub-millisecond, so GC latency is still a significant issue compared to query times.
Go might be acceptable now for niche databases like column-store for certain use cases, though.
Also, see the excellent comment above about distributed systems and server cache issues. You can't do application performance analysis with GC literally everywhere.
The puerile knee-jerk hatred for C on HN has to stop - almost every software you use is written in C, from scripting languages to operating systems to web servers to databases.
Source: DBA who's worked with current databases, as well as a custom database written in Java with significant (ie. brutal) GC problems that required a total refactor (rewrite) to "work" at all.
I'd like to reply specifically to the Go is supposed to be sub-second GC pause latency part - Go's GC is crazy fast, it pauses for only microseconds, comparable to the socket transport overhead. It can't be thought of as Java-like at all.
The fact that database engines written in Java exhibit high latency is not a reason to dismiss all other programming languages except C and C++. It still makes sense to consider modern languages like Rust, Go, Zig, etc.
This is daft. CockroachDB exhibits extremely strong isolation and consistency levels. I believe it is strictly serializable under most circumstances?
Also - anyone who says that x database must be rewritten “because GC” is just making an incredibly un nuanced argument about a nuanced problem. People have built production ready databases in both Java and Go. If you care about low/predictable tail latencies then you have a bunch of other more important problems to solve before you worry about the behaviour of a modern garbage collector. For example: how good is your cache hit ratio? How are your synchronous replication protocols affected by grey failures? That kind of thing.
I believe it only guarantees serializable isolation. You may get strict serializable, but it doesn’t appear to be the case that you will know for sure if any transactions were not linearizable.
No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider.
The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model.
If you spent the last decade writing shitty cross database SQL then you’ll have to silo by customer. That’s the only real constraint.
I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good".
Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).
I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...
Yeah, I probably could have phrased that better. The sibling comment is correct, I meant that if your idea requires abandoning referential integrity or data consistency, it's probably not a good idea. Since it's the database that actually enforces those constraints, it may seem that the database causes the problem. But in most cases, the problem is the data model or the idea itself.
I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"
I would say that most resources would be a mixed bag in most situations but much of the advice skips over something very basic that experienced people tend to assume you know: you are always going to be limited by the number of times you can actually sync a write to a disk and/or how fast you can read back from the disk and these are the main upper bounds. One mistake I see inexperienced engineers make in modern environments is not realizing that in their particular cloud instance they are explicitly or implicitly using the cloud vendor's network storage and they are entirely hamstrung on that. The other is not configuring the database to use all the RAM- but those are usually covered by the existing tuning guides.
The next from a design perspective is to leave natural places to shard/split the data up so you could represent the data consistently on multiple systems. Sometimes it's pretty bonehead things like your database tables related to login are affecting your critical path transactions and/or also tied to the referential integrity of your application data in ways that are hard to split apart. It goes back to the IO thing where if you are reading and writing login or UI traffic you sure as heck are not writing some other critical time sensitive transaction especially if it's to the same disk. The other obvious thing is to be trying to run analytic queries (aka look at many records) on a database tuned for mass INSERT/write traffic and having no plan to do that with a different resource.
If you are still worried about it I would suggest just prototyping it using a currently modern API/RPC server in front of whatever datastore you come up with. As systems have gotten more distributed the straight line speed of connecting to the database directly makes less and less sense in my opinion and having an API server you control allows you to be clever later in ways that might not be obvious today (especially keeping application-aware counters so that you can see what your application is actually doing as query-style logs for databases are extremely expensive in performance). Either way if you sit down and consider the performance characteristics of the underlying system, think about each class of transactions are you considering from your application in a write/read context (hopefully from measured data), and using tools like EXPLAIN you'll quickly start finding what pieces of the puzzle might need to move to start scaling.
The answer also might just be some parts of your application would be better served something other than a database like a message queue, a more straightforward ISAM or key-value store, or depending on your situation just flat files written onto the fastest local storage you can get to then be later transformed and loaded into a database sometime later.
Thanks for giving me some thoughts to turn over in my head. I have to admit, having been able to support most of my projects on a simple LAMP stack, I may simply be overly worried about scaling. But my hope is always to have a sideproject skyrocket.
Did you read the article a little while back about Discord's challenges with storing tons of messages and how they upgraded their tech stack? Did it make sense to you?
The discord post made sense because they understood what their workload was and did the type of 'classifying' queries that I previously described. They also decided to move off Mongo when they were hitting the 'real limits' of the underlying system from not being able to fit critical things in RAM. So they had a solid understanding of what was going on and they moved to rectify it. I don't want to presume to know anything about your sideproject but there's always going to be an limit you hit in any system, and you just need to build monitoring to understand when you are approaching that cliff.
They also had some clear characteristics of their system for their critical query that they could take advantage of: a new message is written to generally once, the messages table scales linearly, and updates and deletes are relatively rare. Using those things they were able to seek out a datastore that helped maximize the thing that's happening the most which is the write, and they were able to engineer a system accordingly. They also know that the messages would grow linearly without bound at a variable rate, so they could plan for that capacity. To me, that makes a lot of sense.
Especially if it is a side project you can't over-engineer capacity but you can do at least the measured math of what actions are actually happening in your system and what are your expectations for those queries. I would suggest that often times you'd want to focus on your critical queries that if slowed down would have the biggest net negative, but you'd have to find those. Every user action in a system should be attributable to some combination of reads or writes that can be reasoned about to try and figure out where there might be problems of contention, linear growth, exponential growth or growth up to a limit for a table, or something else. Either way, once you start classifying some of the critical paths in your system often either an answer or better questions start to come to the top. Without some of this information you can't back of the envelope whether or not your system can take a perceived load.
I'd echo the sibling comment, but also add that thousands a month isn't that much if you have a profitable idea that is internet scale. Thousands a month is the cost of one software engineer.
Sorry, you are absolutely correct that a profitable idea at internet scale can absorb monthly hosting costs of thousands a month. I was using it as a placeholder for "throw money at hardware/cloud services".
Especially if costs can scale with the userbase, it's a very good problem to have.
> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model.
This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because:
1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus)
2. "Trendy" companies in the past decade have been network-type products (social networks, etc), where the structure of the data makes it much harder to silo (people need to follow each other, interact with each other's content, etc)
3. We as an industry took a several-year detour over to NoSQL land as a promised solution to scalability.
Life would be a lot easier if you could say something like:
* I want a Postgres node.
* I'm happy to shard my data by some key (customerId, city, etc) and am willing to accept responsibility for thinking through that sharding key.
* My application has some logic that easily knows which DB to read/write from depending on shard key.
* There's some small amount of "global application data" that might need to live on a single node.
For your "life would be easier" version, isn't that pretty much what Citus does? And Citus is open source so paying for the managed version is optional.
saying "Citus is open source" doesn't mean its as simple as running `yarn install citus; yarn run citus`. Its is much much more complicated than typical open source. You pay for support for a very good reason.
Sharding by customer has a lot of benefits for b2b applications also. You can run each customer in a seperate instance in their own vpc if you want to, and guarantee to your customers that their data is never commingled with another customer. This makes a lot of sense if you have relatively low number relatively high value customers.
It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B.
You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases).
There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.
On paper I’m not dealing with the same problem as the people in this thread, but I can’t help thinking that the root causes are the same.
I have a team that won’t even look at the fact that we measure performance by response time, but we are lumping together write traffic and two classes of read traffic into a single monolithic chunk of code. We are SaaS where our customers write and their customers read. And of course search queries are by far the slowest traffic.
But there’s some sort of mental block about splitting things up that is only slowly changing, and our app is so “flexible” that a load balancer would struggle to tell which urls involve search functionality.
I think people just want to “know where to look” but if your code is on a cluster there already is no “there”. You’re probably looking at some log aggregator anyway, so who gives a tinker’s damn if they come from separate log files on the same box or the same log file?
The mistake is thinking half the system is doing nothing. It’s a bit like saying I’m paying to store my backups offsite but most of the time it’s not really used so it’s a waste of money. It just doesn’t hold.
Most SQL problems that most of us have to deal with stem from inadequate indexes and/or poorly written queries. No fancy active-active setup will ever solve those issues.
Maybe its a bit tortured of a metaphor, but perhaps with continued research we can find a way to use more of the hardware just like how RAID 5 provides more storage over RAID 1 while still keeping parity.
I don't think its foolish to keep asking for more. We still might find it.
If you really want or need to go active-active but also don’t want to think about it much and can live with MySQL my preference would probably be AWS Aurora multi-master if you can live with its limitations:
As noted there and elsewhere though often you are better off just scaling up.
If you do want to really think about it then NewSQL (Cloud Spanner or others) could be the way to go but the cost, performance, compatibility/portability and other concerns make this a much more complex decision that isn’t easy to capture in a blog post (or HN comment ;).
The article is talking about Oracle RAC, and the "crossover" he is talking about was usually really quick Infiniband, not a crossover cable (so, a separate NIC for good reason!). The person that wrote the article doesn't seem to understand RAC enough to actually comment properly. A properly tuned RAC instance will scale horizontally very very well.
Former (recovering?) Exadata user here. I'll go as far as to say that configuring a best-practices RAC cluster is easier than configuring a best-practices MySQL cluster. RAC solves some additional problems, like discovery and load balancing, that MySQL kicks out to the user (e.g. you have to manage DNS and run a tool like ProxySQL separately).
If you look back far enough, it wasn’t infiniband but a small separate fibrechannel fabric, and further back again I recall buying specialist dual-ported SCSI drives that were marketed primarily for cluster i.e. quorum-disk purposes. More than one goat was required to make these configurations work.
(I seem to recall that many SAS drives now are dual-ported but designed as such to fulfil a desire for multi-path resiliency within storage arrays; this may have different characteristics to those needed of a quorum device)
I’ve seen Oracle RAC in use in many banks in particular where uptime and resilience are critical requirements. It scales well horizontally, but at a very high cost.
Databases scale just fine. Its called sharding. Azure has HyperScale (with is Citus), AWS has Aurora. The actual problem with databases is upgrading to major versions without downtime. All the other problems are just noise
Agreed. But one impediment to sharding is bad DB design, which is rife right now because everyone starts with Django/Rails/Laravel and starts treating the DB like a document store that happens to have some relational features.
I think that being able to have separate databases for different domains is pretty good, although it does rely partly on the application layer to keep (UU)IDs consistent.
I'd be curious to know if there's a way to have databases talk to each other to just sync up primary keys for referential integrity. That could maximise the benefit of decoupled databases while still having good referential integrity. And a network disconnection would still mean existing PKs would be in place, it just wouldn't be aware of new ones. Not perfect, but not bad, perhaps.
A dockerized database server is something I cannot understand; I understand bare metal, I can accept virtualized, but I cannot find a good used case for a mid sized or large server (dozens of gigabytes to dozens of terabytes) dockerized and I don't know why a smaller server is a problem.
A docker image is simply packaging. There’s a small, almost always negligible, performance hit. But the value comes in being able to ship a consistent application and configuration pairing. This becomes really valuable when shipping multiple databases for multiple services.
I know what docker does, I never saw a database server deployed in mass. One of my teams manage over 250 database servers, but nobody ever supported the idea of using docker for that (all the servers run on Windows).
Shipping multiple databases for multiple services does not necessarily mean multiple database servers. In the database world I saw large servers with many databases more often than multiple servers of one database each; in the second case it was the vendor laziness (cannot give the name), not than a reasonable business or technical reason. When I asked about it the answer was "we'll consolidate in the next release".
Sure there’s a lot of variables here. If you’re a Windows shop I would not expect a push for docker since it is Linux based. As far as multi-tenant logical databases, that is also highly dependent on workload and failure domains (and significant cost if you’re talking Oracle).
For our team, the reason is standardization. We have our entire baremetal infrastructure running Kubernetes. There is no process for running anything in not-a-Docker-container. This does come with some caveats for stateful databases, but it's still easier to manage than a special-snowflake environment for database deployments.
I have been working on rewriting a monolith into individual services during past year at work - and what we finally implemented for consistency across different databases of services was that we keep a signal queue - where whenever we encounter an inconsistency, a doc is pushed containing the info and we have a corrective service always reading from that queue and doing the necessary updates to tables. We made a heirarchy of source-of-truthness and we use that to decide what to do with inconsistent data. Though, users sometimes have to see a error messagge "We had an error and we are fixing it in few XXX time" based on current queue load - but it has been working fine mostly for us.
Why was that change made? From my understanding users are constantly seeing outages now where they see an error message and have to wait. Were there more outages before? How was the previous system?
just because it's the thing I know best at the moment, it's worth checking out Galera which actually does do multi-master replication. Of course you have to live in MySQL/MariaDB world which is a non-starter for a lot of peeps.
SQLite is a remarkably good solution to most of these problems, if deployed correctly.
For your main line-of-business database? Of course not. But a deployment of rqlite[0] for your service workers in a read-heavy workload? cuts out a round-trip out of the VM, mocking is trivial, there's a lot to like there.
You're missing the whole point of the article. If you set this up in an active-active configuration (if that's even supported) you have the exact same distributed systems problems as the author.
The whole point of having a database is that you need persistent data that is consistent for all clients. You can’t have a bunch of SQLite dbs and get that.
The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas.
A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed a write have their read traffic sent to the leader for the next 10 seconds or so.
Not trivial, but also not impossibly difficult - plenty of places implement this without too much trouble.
Scaling writes is a lot harder - but a well specc'd relational database server will handle tens of thousands of writes per second, so the vast majority of projects will never have to solve this.
When you do need to solve this, patterns for horizontally sharding your data exist. They're not at all easy to implement, but it's not an impossible problem either.
The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box).
Overall, my experience is that the database /enables/ all good ideas. Building stateful applications without a relational database in the mix is usually the wrong choice.
horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas
There's one feature that I'd like to see in that area: partitioned writable replicas. In the same vein that you can partition the table storage across an index, I'd like it to be possible to assign different writers to different parts of a table/database. Of course, you'd still need a single primary replica to handle the transactions that transcend the configured partitions, but we already have routing engines that can transparently redirect an incoming query to any available replica, so it's partly there.
There's probably corner cases lurking that I can't even think of, but in my mind it's the only thing missing from building a truly web-scale (multiple zones, multiple datacenters) ACID-preserving relational database.
This is essentially the idea of VoltDB [1]:
All tables are partitioned and each partition is assigned to a single thread. As long as each query only touches a single partition, you get linear horizontal scalability and serializable isolation. The drawback is that if a query touches multiple partitions, only that one query can be executed on the entire cluster, so this architecture only works for very specific OLTP workloads.
> There's one feature that I'd like to see in that area: partitioned writable replicas.
This is getting more common these days. Google’s Spanner database does this. The database is divided into “tablets” which are each given their own Paxos state. Spanner moves tablets between server instances to do load balancing. There’s no need for you to do the partitioning manually, although if you run into performance problems, you may need to understand how tablets work.
Spanner’s definitely not unique in doing this, but Spanner does have a nice writeup of how it works:
I’ve manually implemented this approach before and I would be a little scared of doing it automatically.
When you do it manually you are very aware of queries that need to go to multiple shards and really do whatever it takes to avoid them.
We sharded by account id and all an individual users data would be on that shard.
Basically the only queries that need to transcend the shard are when you need to enumerate accounts by some non-ID value like an email address. You have to ask all the shards in that case.
Things like creating an account are tricky if you want to make sure that each has a unique email address. You need to use two-phase commit across all the databases to do it reliably.
I think I better title would be “reality reveals that ideas were actually bad”.
The issues described in this blog are about the challenges of persisting a consistent set of data, when you need to support a lot of read and write access. That is just a fundamentally difficult problem. Databases are the most sophisticated tools we have for doing that, but really they just tend to be a set of (hopefully) well implemented trade offs that attempt to nicely balance all of those conflicting requirements.
If a person can’t find a way to adequately persist their data, then their idea is probably bad, or they just don’t have the competence to implement it. An equally suitable title for this post could have been “why doesn’t magic exist”.
Exactly. Consistently and reliably storing business data is something that actually turns out to be important in the real world.
If you can't actually retain your business data, it really doesn't matter how many "good ideas" or fancy deployment strategies for the presentation tier you have.
Man I fucking hate factory boy, by default it fuzzes the input meaning your tests are non deterministic. Honesty, I don’t understand why people use it; how many models do people have where meta programming factory classes makes sense?
I think it comes down to whether you want to maintain a fixture or a factory. In my experience, it can be easy to set up a fixture but over time it increasingly becomes a pain -- sometimes enough that setting up a factory is worth it.
> The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box).
Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD services, doing this vs. pure unit tests that mock the db, yields far better benefit (provided you don't involve heavy volumes in the tests to avoid test data maintenance issues). With this approach, I have found database side edge cases/constraint violations that wouldn't be caught by mocking the database. Also, it is a much more comprehensive test if you take the approach of a) loading an initial state, b) doing business logic mutations and c) compare the final state with the expected state; e.g. you might find side effects that you hadn't considered testing for and so on.
After a bug slipped through because of it, I switched CI to run the same point release of the database, and validate that the schema is identical to production.
If speed is one of the things holding you back from testing with the native database, running the database data on a tmpfs mountpoint is worth looking at.
For me, it is more of zero setup while getting most of the benefits which is "good enough" (tmpfs works great but every new developer would need to do this + setup/permissions needed for docker + differences between Windows/Linux and/or desktop vs CI host etc).
Otherwise I have a single docker-compose.yml file that sets up postgres using "Docker for Desktop" on Mac and Windows and works natively on Linux. CI (Linux) uses the same docker-compose.yml.
I haven't used docker in production, but it has proved very useful in test.
Scaling writes is theoretically really easy via sharding, and I've done it several times. Unfortunately in typical web stacks these days ORMs absolutely do not support sharding in any reasonable way, even if your data model supports it easily (users don't collaborate so you can 100% segregate by user ID and never need cross-shard joins except to static tables).
Agreed re: db mocks, database bugs and quirks should be captured and addressed in tests, not discovered only in production. I don't care how mature the DB you're using is, there's always going to be some edge case behavior that you can't predict until you actually see it happen, even just simple stuff like exactly how an error filters back to the application when you send data that violates a constraint. I've seen way too many tests that make assumptions about these things that turn out to be incorrect, leading to horrible behavior like writes getting through on prod that should have been blocked because of pre-checks that didn't turn out to be enforced the way the test writer assumed they would be.
> The article talks briefly about mocking your database: definitely never do this.
I definitely recommend doing this for same tests: it's the only way to check how your application behaves in case of failures.
The best thing would be to have a DB where you can inject failures, but I'm not aware of any.
So test
- the happy path on the real db
- the sad path on the mock/fake (which might be a light wrapper around a real db, but with the ability of injecting failures)
That's true: the one place where it's a good idea to mock the database is when you want to simulate what would happen if a specific database error occurred that can't be triggered another way.
It's still a fairly new thing for people outside of Google. It's been a few years, so I imagine it's better now...but when I tried CockroachDB, it was pretty easy to trigger horrible write latency. Bad enough to be a non-starter.
I don't recall other than it was before 2.x, but I don't think it's terribly important. The cockroachdb folks were very transparent about it, including a couple of major re-works to address performance. Here's a blog post about one of them: https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/
I was just trying to say that a stable, high performance SQL database with distributed writes is a fairly new thing.
> If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes.
Oh I dunno, I'd still say most claims to be "better" at data integrity with performance, and a lot of it than a good old zOS mainframe with a soundly designed DB2 database on it are [Citation needed]-class...
But yeah, databases certainly aren't the problem here.
I’m not convinced the author knows what they’re talking about. The answer to the question they posed is basically “ACID”, so unclear what all the verbiage is. Also unclear why they expect all readers to have an architecture featuring multiple application servers all sharing a single database, since that is neither classic monolith nor classic microservices.
I get the same impression. Relational with ACID guarantees can't be implemented in a distributed system. Depending on which guarantees you want to give up, you might be able to get close. A lot of newer platforms give up guarantees to achieve something like this.
But if you have multiple nodes that have to sync every transaction over a dedicated NIC, well, that's not a distributed system. It's just multiprocessing with extra latency.
It's a causality problem, not a technology problem.
Information can't be changed in two places simultaneously. That latency, no matter how small, requires us to either eat the latency, give up guarantees, or alter our system's behavior in some other small way.
vitess on k8s gives u some nice options.. sure a little extra application logic but so what. you can easily break out of vitess sharding whereever its needed and you still get the benefit of k8s schuduling.
The issue with scaling the database is that SQL joins (generally) do not scale under CAP.
So, alas, you either go SQL in the early stages and then need to do considerable engineering to down-convert to say, Cassandra or DynamoDB.
Or you accept reduced database language sugar and complexity up-front (no joins, limited index/views, or architect with explicit sharding) with a more scalable database approach.
There's basically no magic sauce for scalable SQL. Frankly most of the people telling you otherwise are selling varying degrees of snake oil.
As people will point out, SQL databases on modern hardware scale pretty freaking large. So you can get a lot of mileage putting off the "true scalability refactor". The good news, by then you should know your queries and data that need full scaling, you aren't guessing ahead of time.
SQL is just a language. Any techniques which can be used scalably (client-side joins for instance) could also be used by an implementation supporting the SQL language. Perhaps your argument holds for some of the more well-known RDBMSes out there, but I don't think SQL necessarily has to be unscalable in the general case.
Let's take the general case of a join: two tables that are keyed/indexed by fundamentally different data.
To distribute data, you have a data distribution scheme. Cassandra and various distributed hash maps (which is the typical approach) it is a consistent hash function. But even if you do distribution using natural ordering, the same problem exists:
The data you are joining is going to be on different nodes on a row-by-row case. Hashing will produce this by the design of the hash function. Natural ordering will do this because different key datatypes will order differently.
In the case of large scale distribution across a LOT of machines (which is what you invariably have to go to once you expend the options in big iron), that means a huge amount of network traffic, with each retrieval needing to be resolved for consistency due to if you want AP. If you rely on CP, then your join is dependent on SO MANY nodes correctly communicating that you become extremely exposed to network partitions, retries, etc.
Thus you either shard your data so all data is on the same machine (but your joins are necessarily subsets of the overall data: only the shard), or you don't and prepare for extremely bad performance, unreliable performance, or approximations of correctness.
>Each tier is either easy to reason about scaling out horizontally except for the database.
Vitess [1], A database clustering system for horizontal scaling of MySQL, or Planetscale [2] which is the SaaS version. Of course everything is good on paper until you run into edge cases. But I am convinced within this decade scaling problem or hassle will be a thing of the past for 95% of us.
I feel like this misses the point. The reason all the other stuff besides the database is so straightforward is because you can localize all the state in the database so you can deal with the hard parts in one place and with maximum support.
The database doesn't ruin all good ideas. It makes all those ideas for other tiers possible.
The article is confusing (confused?), mixing up different concepts like containerisation and active/active database configuration while not even going into detail but only mentioning concepts in passing by. I don't get the point.
165 comments
[ 2.5 ms ] story [ 213 ms ] threadAlso, it seems unfair to say that the database ruins all good ideas when there continues to be significant impressive innovation in the space in cloud services. Like Amazon Aurora for one. If you really want to treat your RDBMS like an elastically scalable (with significant caveats) resource priced by the request, you can.
Also I think these issues arise at the database because that’s were the write concurrency is pushed.
The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem.
Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).
As for the subsecond spikes in latency, these tend to multiply in a distributed system. If serving a client request takes N internal requests, the likelihood of hitting a GC pause somewhere is much larger than if you did only one local request.
Not sure about Go, but none of the "free" Java's GC guarantees low pauses. There is STW fallback even in the most advanced ones like ZGC. So you never know when it stops for more than 1s.
https://blog.golang.org/ismmkeynote
Even if it is a hard guarantee, then, from the link you posted, it is not even generational, so it will scan the whole heap quite frequently, and that is going to influence average performance quite visibly - you definitely dont want a database system to access all its cached memory once in a while.
Database systems are really all about memory and I/O management. You shouldn't outsource those core features to a universal algorithm, unless you wish to forgo any competitive advantage (at least in performance department). So this pushes the devs into the off-heap manual memory management territory, where dragons live (at least in Java, again - maybe Go is better in that regard). I've been there, and I don't recommend.
False. All popular databases are written in C, and continue because of the GC issue. I would not use a general-purpose database written in Java because of GC, for example. We'll see how well Go works in practise.
> sub second spikes in latency
Go is supposed to be sub-second GC pause latency, but understand that most SQL queries are sub-millisecond, so GC latency is still a significant issue compared to query times.
Go might be acceptable now for niche databases like column-store for certain use cases, though.
Also, see the excellent comment above about distributed systems and server cache issues. You can't do application performance analysis with GC literally everywhere.
The puerile knee-jerk hatred for C on HN has to stop - almost every software you use is written in C, from scripting languages to operating systems to web servers to databases.
Source: DBA who's worked with current databases, as well as a custom database written in Java with significant (ie. brutal) GC problems that required a total refactor (rewrite) to "work" at all.
https://blog.golang.org/ismmkeynote
Our transactions implementation is our crown jewel. You might want to check your sources.
Also - anyone who says that x database must be rewritten “because GC” is just making an incredibly un nuanced argument about a nuanced problem. People have built production ready databases in both Java and Go. If you care about low/predictable tail latencies then you have a bunch of other more important problems to solve before you worry about the behaviour of a modern garbage collector. For example: how good is your cache hit ratio? How are your synchronous replication protocols affected by grey failures? That kind of thing.
The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model.
If you spent the last decade writing shitty cross database SQL then you’ll have to silo by customer. That’s the only real constraint.
Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without understanding the problem space is a disaster waiting to happen (but luckily, it may not happen on your watch so you can continue hacking).
I have directly experienced the problems associated with blindly assuming that you can write your own consistency guarantees in Redis. I didn't do it but I had to write some tooling to rewrite the AOF to get some data back...
I don't feel that's what you mean, and it's probably my fault for misreading your comment.
I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"
The next from a design perspective is to leave natural places to shard/split the data up so you could represent the data consistently on multiple systems. Sometimes it's pretty bonehead things like your database tables related to login are affecting your critical path transactions and/or also tied to the referential integrity of your application data in ways that are hard to split apart. It goes back to the IO thing where if you are reading and writing login or UI traffic you sure as heck are not writing some other critical time sensitive transaction especially if it's to the same disk. The other obvious thing is to be trying to run analytic queries (aka look at many records) on a database tuned for mass INSERT/write traffic and having no plan to do that with a different resource.
If you are still worried about it I would suggest just prototyping it using a currently modern API/RPC server in front of whatever datastore you come up with. As systems have gotten more distributed the straight line speed of connecting to the database directly makes less and less sense in my opinion and having an API server you control allows you to be clever later in ways that might not be obvious today (especially keeping application-aware counters so that you can see what your application is actually doing as query-style logs for databases are extremely expensive in performance). Either way if you sit down and consider the performance characteristics of the underlying system, think about each class of transactions are you considering from your application in a write/read context (hopefully from measured data), and using tools like EXPLAIN you'll quickly start finding what pieces of the puzzle might need to move to start scaling.
The answer also might just be some parts of your application would be better served something other than a database like a message queue, a more straightforward ISAM or key-value store, or depending on your situation just flat files written onto the fastest local storage you can get to then be later transformed and loaded into a database sometime later.
Did you read the article a little while back about Discord's challenges with storing tons of messages and how they upgraded their tech stack? Did it make sense to you?
They also had some clear characteristics of their system for their critical query that they could take advantage of: a new message is written to generally once, the messages table scales linearly, and updates and deletes are relatively rare. Using those things they were able to seek out a datastore that helped maximize the thing that's happening the most which is the write, and they were able to engineer a system accordingly. They also know that the messages would grow linearly without bound at a variable rate, so they could plan for that capacity. To me, that makes a lot of sense.
Especially if it is a side project you can't over-engineer capacity but you can do at least the measured math of what actions are actually happening in your system and what are your expectations for those queries. I would suggest that often times you'd want to focus on your critical queries that if slowed down would have the biggest net negative, but you'd have to find those. Every user action in a system should be attributable to some combination of reads or writes that can be reasoned about to try and figure out where there might be problems of contention, linear growth, exponential growth or growth up to a limit for a table, or something else. Either way, once you start classifying some of the critical paths in your system often either an answer or better questions start to come to the top. Without some of this information you can't back of the envelope whether or not your system can take a perceived load.
Especially if costs can scale with the userbase, it's a very good problem to have.
This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because:
1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus)
2. "Trendy" companies in the past decade have been network-type products (social networks, etc), where the structure of the data makes it much harder to silo (people need to follow each other, interact with each other's content, etc)
3. We as an industry took a several-year detour over to NoSQL land as a promised solution to scalability.
Life would be a lot easier if you could say something like:
* I want a Postgres node.
* I'm happy to shard my data by some key (customerId, city, etc) and am willing to accept responsibility for thinking through that sharding key.
* My application has some logic that easily knows which DB to read/write from depending on shard key.
* There's some small amount of "global application data" that might need to live on a single node.
There’s YugabyteDB.
It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B.
You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases).
There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.
I have a team that won’t even look at the fact that we measure performance by response time, but we are lumping together write traffic and two classes of read traffic into a single monolithic chunk of code. We are SaaS where our customers write and their customers read. And of course search queries are by far the slowest traffic.
But there’s some sort of mental block about splitting things up that is only slowly changing, and our app is so “flexible” that a load balancer would struggle to tell which urls involve search functionality.
I think people just want to “know where to look” but if your code is on a cluster there already is no “there”. You’re probably looking at some log aggregator anyway, so who gives a tinker’s damn if they come from separate log files on the same box or the same log file?
Row based permissions are not great imo, either performance is crap or you have some weird bug.
Most SQL problems that most of us have to deal with stem from inadequate indexes and/or poorly written queries. No fancy active-active setup will ever solve those issues.
I don't think its foolish to keep asking for more. We still might find it.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
As noted there and elsewhere though often you are better off just scaling up.
If you do want to really think about it then NewSQL (Cloud Spanner or others) could be the way to go but the cost, performance, compatibility/portability and other concerns make this a much more complex decision that isn’t easy to capture in a blog post (or HN comment ;).
(I seem to recall that many SAS drives now are dual-ported but designed as such to fulfil a desire for multi-path resiliency within storage arrays; this may have different characteristics to those needed of a quorum device)
I'd be curious to know if there's a way to have databases talk to each other to just sync up primary keys for referential integrity. That could maximise the benefit of decoupled databases while still having good referential integrity. And a network disconnection would still mean existing PKs would be in place, it just wouldn't be aware of new ones. Not perfect, but not bad, perhaps.
Shipping multiple databases for multiple services does not necessarily mean multiple database servers. In the database world I saw large servers with many databases more often than multiple servers of one database each; in the second case it was the vendor laziness (cannot give the name), not than a reasonable business or technical reason. When I asked about it the answer was "we'll consolidate in the next release".
For your main line-of-business database? Of course not. But a deployment of rqlite[0] for your service workers in a read-heavy workload? cuts out a round-trip out of the VM, mocking is trivial, there's a lot to like there.
[0]: https://github.com/rqlite/rqlite
A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed a write have their read traffic sent to the leader for the next 10 seconds or so.
Not trivial, but also not impossibly difficult - plenty of places implement this without too much trouble.
Scaling writes is a lot harder - but a well specc'd relational database server will handle tens of thousands of writes per second, so the vast majority of projects will never have to solve this.
When you do need to solve this, patterns for horizontally sharding your data exist. They're not at all easy to implement, but it's not an impossible problem either.
The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box).
Overall, my experience is that the database /enables/ all good ideas. Building stateful applications without a relational database in the mix is usually the wrong choice.
There's one feature that I'd like to see in that area: partitioned writable replicas. In the same vein that you can partition the table storage across an index, I'd like it to be possible to assign different writers to different parts of a table/database. Of course, you'd still need a single primary replica to handle the transactions that transcend the configured partitions, but we already have routing engines that can transparently redirect an incoming query to any available replica, so it's partly there.
There's probably corner cases lurking that I can't even think of, but in my mind it's the only thing missing from building a truly web-scale (multiple zones, multiple datacenters) ACID-preserving relational database.
[1] https://www.voltdb.com/
This is getting more common these days. Google’s Spanner database does this. The database is divided into “tablets” which are each given their own Paxos state. Spanner moves tablets between server instances to do load balancing. There’s no need for you to do the partitioning manually, although if you run into performance problems, you may need to understand how tablets work.
Spanner’s definitely not unique in doing this, but Spanner does have a nice writeup of how it works:
https://static.googleusercontent.com/media/research.google.c... (PDF)
Thus the write throughput can be very high but the write latency is also very high.
When you do it manually you are very aware of queries that need to go to multiple shards and really do whatever it takes to avoid them.
We sharded by account id and all an individual users data would be on that shard.
Basically the only queries that need to transcend the shard are when you need to enumerate accounts by some non-ID value like an email address. You have to ask all the shards in that case.
Things like creating an account are tricky if you want to make sure that each has a unique email address. You need to use two-phase commit across all the databases to do it reliably.
If you're happy letting the database do that for you, that's how current CockroachDB works.
The issues described in this blog are about the challenges of persisting a consistent set of data, when you need to support a lot of read and write access. That is just a fundamentally difficult problem. Databases are the most sophisticated tools we have for doing that, but really they just tend to be a set of (hopefully) well implemented trade offs that attempt to nicely balance all of those conflicting requirements.
If a person can’t find a way to adequately persist their data, then their idea is probably bad, or they just don’t have the competence to implement it. An equally suitable title for this post could have been “why doesn’t magic exist”.
If you can't actually retain your business data, it really doesn't matter how many "good ideas" or fancy deployment strategies for the presentation tier you have.
I honestly thought you meant to never ‘ridicule’ your database, and I was thinking “that might be a healthy level of respect” :)
Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD services, doing this vs. pure unit tests that mock the db, yields far better benefit (provided you don't involve heavy volumes in the tests to avoid test data maintenance issues). With this approach, I have found database side edge cases/constraint violations that wouldn't be caught by mocking the database. Also, it is a much more comprehensive test if you take the approach of a) loading an initial state, b) doing business logic mutations and c) compare the final state with the expected state; e.g. you might find side effects that you hadn't considered testing for and so on.
It is easy enough to setup and tear down using docker.
https://vladmihalcea.com/how-to-run-integration-tests-at-war...
Otherwise I have a single docker-compose.yml file that sets up postgres using "Docker for Desktop" on Mac and Windows and works natively on Linux. CI (Linux) uses the same docker-compose.yml.
I haven't used docker in production, but it has proved very useful in test.
https://github.com/zonkyio/embedded-database-spring-test
also helps with having it verify all your database migrations as well
Agreed re: db mocks, database bugs and quirks should be captured and addressed in tests, not discovered only in production. I don't care how mature the DB you're using is, there's always going to be some edge case behavior that you can't predict until you actually see it happen, even just simple stuff like exactly how an error filters back to the application when you send data that violates a constraint. I've seen way too many tests that make assumptions about these things that turn out to be incorrect, leading to horrible behavior like writes getting through on prod that should have been blocked because of pre-checks that didn't turn out to be enforced the way the test writer assumed they would be.
I definitely recommend doing this for same tests: it's the only way to check how your application behaves in case of failures.
The best thing would be to have a DB where you can inject failures, but I'm not aware of any. So test - the happy path on the real db - the sad path on the mock/fake (which might be a light wrapper around a real db, but with the ability of injecting failures)
If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes.
Databases are not the problem.
I was just trying to say that a stable, high performance SQL database with distributed writes is a fairly new thing.
Oh I dunno, I'd still say most claims to be "better" at data integrity with performance, and a lot of it than a good old zOS mainframe with a soundly designed DB2 database on it are [Citation needed]-class...
But yeah, databases certainly aren't the problem here.
But if you have multiple nodes that have to sync every transaction over a dedicated NIC, well, that's not a distributed system. It's just multiprocessing with extra latency.
CockroachDB and YugabyteDB would like to have a word.
Information can't be changed in two places simultaneously. That latency, no matter how small, requires us to either eat the latency, give up guarantees, or alter our system's behavior in some other small way.
Fauna, Upstash, DynamoDB, CosmoDB, Firestore, Planetscale.
So, alas, you either go SQL in the early stages and then need to do considerable engineering to down-convert to say, Cassandra or DynamoDB.
Or you accept reduced database language sugar and complexity up-front (no joins, limited index/views, or architect with explicit sharding) with a more scalable database approach.
There's basically no magic sauce for scalable SQL. Frankly most of the people telling you otherwise are selling varying degrees of snake oil.
As people will point out, SQL databases on modern hardware scale pretty freaking large. So you can get a lot of mileage putting off the "true scalability refactor". The good news, by then you should know your queries and data that need full scaling, you aren't guessing ahead of time.
To distribute data, you have a data distribution scheme. Cassandra and various distributed hash maps (which is the typical approach) it is a consistent hash function. But even if you do distribution using natural ordering, the same problem exists:
The data you are joining is going to be on different nodes on a row-by-row case. Hashing will produce this by the design of the hash function. Natural ordering will do this because different key datatypes will order differently.
In the case of large scale distribution across a LOT of machines (which is what you invariably have to go to once you expend the options in big iron), that means a huge amount of network traffic, with each retrieval needing to be resolved for consistency due to if you want AP. If you rely on CP, then your join is dependent on SO MANY nodes correctly communicating that you become extremely exposed to network partitions, retries, etc.
Thus you either shard your data so all data is on the same machine (but your joins are necessarily subsets of the overall data: only the shard), or you don't and prepare for extremely bad performance, unreliable performance, or approximations of correctness.
Vitess [1], A database clustering system for horizontal scaling of MySQL, or Planetscale [2] which is the SaaS version. Of course everything is good on paper until you run into edge cases. But I am convinced within this decade scaling problem or hassle will be a thing of the past for 95% of us.
[1] https://vitess.io
[2] https://www.planetscale.com
The database doesn't ruin all good ideas. It makes all those ideas for other tiers possible.