Haha, I totally hear you. But but, we didn't really build the raft consensus layer from scratch. We used an existing robust library for that: https://github.com/baidu/braft
Ah, sure: I did not consider Redis at all. My goal in the Explore phase was to keep the data in the same process as my code, and replacing MySQL with any other database doesn't really help here. This was a developer-productivity goal, not a performance goal.
No I haven’t because it’s quite complicated. Databases are very much a solved problem. Unfortunately, this architecture is going to be nigh impossible to hire for and when it goes absolutely sideways recovery will be difficult.
Compared to installing, configuring and maintaining an installation of Redis, this absolutely is complicated. Do you think this is less complicated than using Redis?
In what way is setting up redis or writing a program yourself P hard? What’s the input that leads to polynomial time? And what kind of metric is that? If setting up redis takes me one day or I can write a software myself in a month, does it matter if both are P hard?
And if you have an hourly wage over $1, I am very sure that redis is cheaper at the end of the day than programming your own software and using that.
Polynomial time means that both are deterministic. The diffreence between the two only comes down to how much has to type and copy and paste, provided that the person is well aware and experienced to do both. And the total time for either is negligible, while Raft saves you more money long termin infra costs.
The argument that im fighting agaist is that when someone says its more complex, what they mean is that they dont have experience in doing that. From a business perspective, this is something to consider when hiring from.an average pool, since you point about salary is correct, but the assumption that every single engineer fits this criteria is not correct.
Redis is best as an in-memory cache, not a database. Having used it in production for roughly a decade, I don't trust it's on-disk capabilities (AOF/RDB etc) as either solid or reliable (or even performant) in an emergency scenario, especially with DR or DB migration in mind.
FWIW, how I read the article it was just an implementation of the original Redis, but with some other language (and this types) than Tcl.
Redis is/was basically just Tcl-typed which are persisted to disk using snapshots (Tcl commands) and append-only Tcl commands, that had a network protocol for non-Tcl applications to talk to
Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database.
That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications.
I think even SQLite itself wasn't as ubiquitous (edit: it didn't exist) when pg write viaweb. If SQLite wasn't there and my options were basically key value stores, I could as well use filesystem in most cases.
Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate.
We also got advanced features like FTS which are useful for desktop and mobile apps.
Today it's a good choice to use RDBMS for desktop apps.
> If SQLite wasn't there and my options were basically key value stores
Well, there were "options" other than KV stores - MySQL launched a month before Viaweb (but flakey for a good long while.) Oracle was definitely around (but probably $$$$.) mSQL was being used on the web and reasonably popular by 1995 (cheap! cheerful! not terrible!)
(definitely understand making your own in-memory DB in 1995 though)
if pg is still stuck in the 90s lisp, if bet it's just a single process with the site in ram, using make-object-persistent and loading as needed (kinda like python pickle).
It just persists its in-memory data structures to disk. Here's the source of an old version; note uses of `diskvar` and `disktable`. A "table" here is just a hashtable.
I think that is just written to disk as something like file41207393 when you click reply.
When the system needs an item it sees if it's cached in memeory and otherwise reads it from disk and I think that is pretty much the whole memory system. Some other stuff like user id that works in the same sort of way.
I was certainly inspired by PG's writing (after all we do use Common Lisp, and it's hard to avoid PG in this space). But I don't think they did things like transaction logs like how bknr.datastore does, which makes the development process a lot more seamless.
> Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0].
After reading the link, I don't think that database means the same thing for everyone.
The vwfaq still mentions loading data from disk, and also mention "start up a process to respond to an HTTP request." This suggests that by "database" they meant a separate server dedicated to persist data, and having to communicate with another server to fetch that data.
Obviously, this leaves SQLite out of this definition of database. Also, if you're loading data from disk already, either you're using a database or you're implementing your own ad-hoc persistence layer. Would you still consider you're using a database if you load data from SQLite at app start?
The problem with this sort of mental model is that it ignores the fact that the whole point of a database is to persist and fetch data in a way that is convenient to you without having to bother about low-level details. Storing data in a database does not mean running a postgres instance somewhere and fetching data over the web. If you store all your data in-memory and have a process that saves snapshots to disk using a log-structured data structure... Congratulations, you just developed your own database.
it was a different time. to my knowledge, viaweb was a series of common lisp instances. All states for a user session was held IN MEMORY on the individual machine. I remember reading somewhere that they would be on a call with a user on production and patch bugs in real time while they were on the phone.
The web has gotten bigger and a lot of these practices simply would not fly today. If I was pushing a live fix on our prod machine with the amount of testing doing it live while on the customer is on the phone entails today, a good portion of you would be questioning my sanity.
An important reason that practice wasn't as reckless as it sounds is that early Viaweb was just a page builder. The actual web stores its customers were building were static HTML, so updating a customer's instance while talking to them on the phone only affected that one user's backend.
Not sure I would call that setup simple, but it is interesting. I have honestly never heard of ‘Raft’ or the Raft Consensus Protocol or bknr.datastore, so always happy to learn something on a Friday night.
I agree, the infrastructure required to make this happen eventually gets quite complicated. But the developer experience is what's super simple. If somebody had to take all our infrastructure and just use it to build their next big app, they can get the simplicity without worrying about the internal plumbing.
Raft is fantastic and most modern systems with more than one node are built on Raft. It is actually proven to be equivalent to Paxos, but the semantics of it are closer to what you would prefer as a software writer and the implementation is much simpler.
What they described early on in the article was basically how NUMA machines worked (eg SGI Altix or UV). Also, their claimed benefit was being able to parallelize things with multithreading in low-latency, huge RAM. Clustering came as a low-cost alternative to $1+ million machines. There’s similarities to persistence in AS/400, too, where apps just wrote memory that gets transparently mapped to disk.
Now, with cheap hardware, they’re going back in time to the benefits of clustered, NUMA machines. They’ve improved on it along the way. I did enjoy the article.
Another trick from the past was eliminating TCP/IP stacks from within clusters to knock out their issues. Solutions like Active Messages were a thin layer on top of the hardware. There’s also designs for network routers that have strong consistency built into them. Quite a few things they could do.
If they get big, there’s hardware opportunities. On CPU side, SGI did two things. Their NUMA machines expanded the number of CPU’s and RAM for one system. They also allowed FPGA’s to plug directly into the memory bus to do custom accelerators. Finally, some CompSci papers modified processor ISA’s, networks on a chip, etc to remove or reduce bottlenecks in multithreading. Also, chips like OpenPiton increase core counts (eg 32) with open, customizable cores.
My first thought was, “oh, I used to do this when I wrote Common Lisp, it’s funny someone rediscovered that technique in <rust/typescript/java/whatever>”.
This is cool! I’m always excited by people trying simpler things, as a big fan of using Boring Technology.
But I have some bad news: you haven’t built a system without a database, you’ve just built your own database without transactions and weak durability properties.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk.
This is actually not an easy thing to do. If your shutdowns are always clean SIGSTOPs, yes, you can reliably flush writes to disk. But if you get a SIGKILL at the wrong time, or don’t handle an io error correctly, you’re probably going to lose data. (Postgres’ 20-year fsync issue was one of these: https://archive.fosdem.org/2019/schedule/event/postgresql_fs...)
The open secret in database land is that for all we talk about transactional guarantees and durability, the reality is that those properties only start to show up in the very, very, _very_ long tail of edge cases, many of which are easily remedied by some combination of humans getting paged and end users developing workarounds (eg double entry bookkeeping). This is why MySQL’s default isolation level can lose writes: there are usually enough safeguards in any given system that it doesn’t matter.
A lot of what you’re describing as “database issues” problem don’t sound to me like DB issues, so much as latency issues caused by not colocating your service with your DB. By hand-rolling a DB implementation using Raft, you’ve also colocated storage with your service.
> Screenshotbot runs on their CI, so we get API requests 100s of times for every single commit and Pull Request.
I’m sorry, but I don’t think this was as persuasive as you meant it to be. This is the type of workload that, to be snarky about, I could run off my phone[0]
> This is actually not an easy thing to do. If your shutdowns are always clean SIGSTOPs, yes, you can reliably flush writes to disk. But if you get a SIGKILL at the wrong time, or don’t handle an io error correctly, you’re probably going to lose data.
Thanks for the comment! This is handled correctly by Raft/Braft. With Raft, before a transaction is considered committed it must be committed by a majority of nodes. So if the transaction log gets corrupted, it will restore and get the latest transaction logs from the other node.
> I’m sorry, but I don’t think this was as persuasive as you meant it to be.
I wasn't trying to be persuasive about this. :) I was trying to drive home the point that you don't need a massively distributed system to make a useful startup. I think some founders go the opposite direction and try to build something that scales to a billion users before they even get their first user.
Wait, so you’re blocking on a Raft round-trip to make forward progress? That’s the correct decision wrt durability, but…
I’m now completely lost as to why you believe this was a good idea over using something like MySQL/Postgres/Aurora. As I see it, you’ve added complexity in three different dimensions (novel DB API, novel infra/maintenance, and novel oncall/incident response) with minimal gain in availability and no gain in performance. What am I missing?
(FWIW, I worked on Bigtable/Megastore/Spanner/Firestore in a previous job. I’m pretty familiar with what goes into consensus, although it’s been a few years since I’ve had to debug Paxos.)
> I was trying to drive home the point that you don't need a massively distributed system to make a useful startup. I think some founders go the opposite direction and try to build something that scales to a billion users before they even get their first user.
This reads to me as exactly the opposite: overengineering for a problem that you don’t have.
For exactly the reasons you describe, I would argue the burden of proof is on you to demonstrate why Redis, MySQL, Postgres, SQLite, and other comparable options are insufficient for your use case.
To offer you an example: let’s say your Big Customer decides “hey, let’s split our repo into N micro repos!” and they now want you to create N copies of their instance so they can split things up. As implemented, you’ll now need to implement a ton of custom logic for the necessary data transforms. With Postgres, there’s a really good chance you could do all of that by manipulating the backups with a few lines of SQL.
> As implemented, you’ll now need to implement a ton of custom logic for the necessary data transforms. With Postgres, there’s a really good chance you could do all of that by manipulating the backups with a few lines of SQL.
Isn’t writing «a few Lines of SQL» also custom logic? The difference is just the language.
It is also possible that the custom data store is more easily manipulated with other languages than SQL.
SQL really is great for manipulating data, but not all relational databases are easy to work with.
> Wait, so you’re blocking on a Raft round-trip to make forward progress? That’s the correct decision wrt durability, but…
Yeah. I hope it was clear in my post that the goal was developer productivity, not performance.
The round trip is only an issue on writes, and reads are super fast. At least in my app, this works out great. The writes also parallelize nicely with respect to the round trips, since the underlying Raft library just bundles multiple transactions together. Where it is a bottleneck is if you're writing multiple times sequentially on the same thread.
The solution there is you create a single named transaction that does the multiple writes. Then the only thing that needs to be replicated is that one transaction even though you might be writing multiple fields.
> it’s been a few years since I’ve had to debug Paxos
And this is why I wouldn't have recommended this with Paxos. Raft on the other hand is super easy for anyone to understand.
Seems weird to start with “not talking about using something like SQLite where your data is still serialized”, then end up with a home grown transaction log that requires serialization and needs to be replicated, which is how databases are replicated anyway.
If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your database”. If your data fits entirely in RAM, then use a ramdisk for the database if you want, and replicate it to permanent storage with standard tools. Now that’s actually simple.
I do feel like this largely summarizes as "we built our own sqlite + raft replication", yeah. But without sqlite's battle-tested reliability or the ability to efficiently offload memory back to disk.
So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar.
That overly-simplified summary aside, I quite like the idea and I think the post does a pretty good job of selling the concept. For a lot of systems it'll scale more than well enough to handle most or all of your business even if you become abnormally successful, and the performance will be absurdly good compared to almost anything else.
- In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ?
- I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.
It depends on what kind of transaction support you want. If your transactions need to span rqlite API requests then no, rqlite doesn't support that (due to the stateless nature of HTTP requests). That sort of thing could be developed, but it's substantial work. I have some design ideas, it may arrive in the future.
If you need to ensure that a given API request (which can contain multiple SQL statements) is atomically processed (all SQL statements succeed or none do) that is supported however [1]. That's why I think of rqlite as closer to the kind of use cases that etcd and Consul support, rather than something like Postgres -- though some people have replaced their use of Postgres with rqlite! [2]
Thank you - so my takeaway is that rqlite is well suited for distributed “publishing” of data ala etcd, but it is possible to use it as a Postgres replacement - thank you I will give it a go
As for your second question, I don't think you'd benefit much from than that, for two reasons:
- rqlite is a Raft based system, with quorum requirements. Running 2-node systems don't make much sense. [1]
- Secondly, all writes go to the Raft leader (rqlite makes sure this happens transparently if you don't initially contact the Leader node [2]). A load balancer, in this case, isn't going to allow you to "spread load". What is load balancer is useful for when it comes to rqlite is making life simpler for clients -- they just hit the load balancer, and it will find some rqlite node to handle the request (redirecting to the Leader if needed).
I'll throw in a "ehh... sorta" though rqlite is quite neat and very much worth considering.
The main caveat here is that rqlite is an out-of-process database, which you communicate with over http. That puts it on similar grounds as e.g. postgres, just significantly lighter weight, and somewhat biased in favor of running it locally on every machine that needs the data.
So minimum read latency is likely much lower than postgres, but it's still noticeable when compared to in-process stuff, and you lose other benefits of in-process sqlite, like trivial extensibility.
They basically only save on serialization & deserialization at query time, which I would consider an infinitesimal saving in the vast majority of use cases. They claim to be able to build some magical index that's not possible with existing disk-based databases (I didn't read the linked blog post). They lose access to a nice query language and entire ecosystems of tools and domain knowledge.
I fail to see how this little bit of saving justifies all the complexity for run-of-the-mill web services that fit on one or a few servers as described in the article. The context isn't large scale services where 1ms/request saving translates to $$$, and the proposal doesn't (vertically) scale anyway.
You should probably RTFA before making broad assumptions on their solution and how it works. Most of what you wrote is both incorrect and addressed in the article.
Telling people to RTFA is against site guidelines. And I read the entire article before making this comment. If you think I’m wrong, you reply with what’s wrong, not some useless “you’re wrong, RTFA”.
The only thing in my comment that’s not directly based on the article is a handwavy 1ms/request saving estimate, and since they don’t provide any measurement, it’s anyone’s guess.
Is telling the people to RTFA against the guidelines?
The guideline specifically advises to do what GP did: Instead of commenting whether or not someone read the article, to tell them that article answers their questions.
SQlite doesn't do Raft. There isn't any simple way to do replicated SQlite. (In fact, writing your own database is probably the simplest way currently, if SQlite+Raft is actually what you want.)
Agreed. Reinventing the WAL means reinventing (or ignoring) all the headaches that come with it. I got the impression it takes them a long time to recover from the logs, so they likely haven't even gotten as far as log checkpointing.
> Agreed. Reinventing the WAL means reinventing (or ignoring) all the headaches that come with it.
But if the blogger learned SQLite, how would they have a topic to blog about?
Also, no benchmarks. It's quite odd that an argument grounded on performance claims does not bother to put out any hard data comparing the output of this project. I'm talking about basic things like how does this contrived custom ad-hoc setup compare with vanilla, out-of-the-box SQLite deployment? Which one performs worse and by how much? How does the performance difference reflect in request times and infrastructure cost? Does it actually pay off to replace the dozen lines of code of on boarding SQLite with a custom, in-development, ad-hoc setup? I mean, I get the weekend personal project vibe of this blog post, but if this is supposed to be a production-minded project then step zero would have been a performance test on the default solution. Where is it?
> It's quite odd that an argument grounded on performance claims
I probably did a bad job then, because everything in the blog post was meant to be developer productivity claims, not performance claims. (I come from a developer productivity background. I'm decent at performance stuff, but it's not what excites me, since for most companies my size performance is not critical as long as it scales.)
Also the OS will cache a lot of the reads even if your database isn’t sophisticated enough or tuned correctly. Still could be a fun exercise, as with all things on here.
I used to work on a telecom platform (think something that runs 4G services), where every node was just part of an in-memory database that replicated using 2PC and just did periodic snapshot to avoid losing data. Basically processes were colocated with their data in the DB.
I worked on a lottery / casino system that was similar. In memory database ( memory mapped files), with a WAL log for transaction replay / recovery. There was also a periodic snapshot capability. It was incredibly low latency on late 90's era hardware.
Very erlang/otp. Joe Armstrong used to rant to anyone who would listen that we used databases too often. If data was important, multiple nodes probably need a copy of it. If multiple nodes need a copy, you probably have plenty of durability.
Even if you weren't using erlang, his influence (and in general, ericsson) permeates the telecom industry.
Setting up a single server with database replication and restore functionality is arguably more complex then setting this up.
There are libraries available to wrap your stuff with this algorithm, and the benefit is that you write your server like it would run on a single machine, and then when launching it in prod across multiple, everything just works.
I think it's important to understand that every startup goes through three phases: Explore, Expand, Extract. What's simple in one phase isn't simple in the other.
A transactional database is simple in Expand and Extract, but adds additional overhead during the Explore phase, because you're focusing on infrastructure issues rather than product. Data reliability isn't critical in the Explore phase either, because you just don't have customers, so you just don't have data.
Having everything in memory with bknr.datastore (without replication) is simple in the Explore phase, but once you get to Expand phase it adds operational overhead to make sure that data is consistent.
But by the time I've reached the Expand phase, I've already proven my product and I've already written a bunch of code. Rewriting it with a transactional database doesn't make sense, and it's easier to just add replication on top of it with Raft.
I'd assume in the beginning you do not want to spend time writing a bunch of highly difficult code until you've proven your idea/product. Then when you're big enough and have the money, start replacing things where it makes sense. It seems to be the strategy used by many companies.
Unless, of course, your startup is in the business of selling DBMSes.
Absolutely. By the way, if it wasn't clear from my blog post, in the Explore phase, I used an existing library to do this. It was only in the Expand phase that I put this existing library behind a Raft replication.
Having Explored with a transactional database: I really can't agree. Just change your database, migrations are easy and should be something you're comfortable doing at any time, or you'll get stuck working around it for 100x more effort in the future.
That was the biggest disconnect I had as well. SQL db have the _best_ data migration tooling and practices of any data system. It’s not addressed in the article how migrations are handled with this system but I’m assuming it’s a hand rolled set of code for each one.
I think sql db make the most sense during the explore phase and you switch off of them once you know you need an improvement somewhere (like latency or horizontal scalability).
And this comes to the difference between Explore phase and Expand phase.
In the Explore phase, data migration was just running code on the production server via a REPL. Some migrations such as adding/removing fields are just a matter of hot-reloading code in CL, so there weren't a lot of migrations that we had to manually run.
In the Expand phase, once you add replication, this does become hard and we did roll out our own migration framework. But by this point we already have a lot of code built out, so we weren't going to replace it with a transactional database.
Essentially, we optimized for the Explore phase, and "dealt with the consequences" in the Expand phase (but the consequences aren't as bad as you might think).
I’ve done tons on traditional application server + database on the same server projects. There’s zero infrastructure issue there. You keep implying that a not-in-process RDBMS has to be its own server and that’s super strange. Not to mention having a separate db server also doesn’t add much overhead at all in the early stage, even if you’re doing it for the very first time (been there, done that).
I get the desire to experiment with interesting things, but it seems like such a huge waste of time to avoid having to learn the most basic aspects of MySQL or postgres. You could "just" build on top of and be done with it, especially if you're running in a public cloud provider. I don't buy the increased RTT or troubles with concurrency issues, the latter having simple solutions by basic tuning, or breaking out your noisy customers. There's another post on their blog mentioning the possibility of adding 10 million rows per day and the challenges of indexing that. That's... literally nothing and I don't think even 10x that justifies having to engineer a custom solution.
Worse is better until you absolutely need to be less worse, then you'll know for sure. At that point you'll know your pain points and can address them more wisely than building more up front.
> I get the desire to experiment with interesting things, but it seems like such a huge waste of time to avoid having to learn the most basic aspects of MySQL or postgres.
For server-based database engines you can still make an argument on shedding network calls. It's dubious, but you can.
What's baffling is that the blogger tries to justify not picking up SQLite claiming it might have features that they don't need, which is absurd and does not justify anything.
The blog post reads like a desperate attempt to start with a poor solution to a fictitions problem and proceed to come up with far-fetched arguments hoping to reject the obvious solution.
If you want to shed network calls, the easiest solution would be to just run postgres or MySql on the same server and connecting to it via Unix domain socket. So even if SQLite wasn't an option network overhead isn't a good argument
Here’s the thing that I wonder about: would their business be successful if they didn’t spend all this time reinventing the wheel? Just by building it out in the open and blogging about it, they popularize their product and show their technical prowess. If they’d use the boring technologies that one sticks together and all works, they’d have less to talk about—and thus less publicity?
Wondering if my thinking is flawed, or if going this—arguably unnecessary—extra mile is part of the product and being successful in the space.
[Author here] The transactions and snapshots are still logged to disk. So if the cluster goes down and comes back up, each one just reloads the state. Until at least two machines are back up, we won't be able to serve requests though.
Not sure what you mean by ephemeral things. If you mean things like file descriptors, they are not stored. Technically the snapshot is not a simple snapshot of RAM, it snapshots through all the objects in memory that are set up to be part of the datastore. (It's a bit more complicated and flexible than this, but that's the general idea.)
Hmm, but the problem with having in-memory objects rather than a db is you end up having to replicate alot of the features of a relational database to get a usable system. And adding all these extra features you want from those dbs end up making a simple solution not very simple at all.
To some extent I think this is an "if all you have is a hammer..." situation. Relational DBs are often not a great fit for how contemporary software manages data in memory (hence the proliferation of ORMs, and adapter layers like graphql). I think it's often easier to write out one's relations in the data structures directly, rather than mapping them to queries and joins
To clarify, as I think some people have misunderstood: we used an existing library called bknr.datastore to handle the "database" part of the in-memory store, so we didn't have to invent too much. Our only innovation here was during the Expand phase, where we put that datastore behind a Raft replication.
But why, when you can build things in an ordinary way with ordinary tech like Python/Java/C#/TypeScript and Postgres. Lots of developers know it, lots of answers to your questions online, the AI knows how to write it.
Reading posts like this makes me think the founders/CTO is mixing hobby programming with professional programming.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. So if you have a line like foo.setBar(2), this will first write a transaction that says we’ve changed the bar field of foo to 2, and then actually set the field to 2. An operation like new Foo() writes a transaction to disk to say that a Foo object was created, and then returns the new object.
>
> And so, if your process crashes and restarts, it first reloads the snapshot, and replays the transaction logs to fully recover the state. (Notice that index changes don’t need to be part of the transaction log. For instance if there’s an index on field bar from Foo, then setBar should just update the index, which will get updated whether it’s read from a snapshot, or from a transaction.)
That’s a database. You even linked to the specific database you’re using [0], which describes itself as:
> periodically just take a snapshot of everything in RAM.
Sound similar to `stop the world Garbage collection` in Java. Does your entire processing comes to halt when you do this? How frequently do you need to take snapshots? Or do you have a way to do this without halting everything
Good catch! Snapshotting was certainly a bottleneck that I chose not to write about.
But we aren't really taking the snapshot of RAM, instead we're running some code asking each object to snapshot itself into a stream. If you do this naively, it will block writes on the server until the snapshot is done (reads will continue to work).
But Raft has a protocol for asynchronous snapshots. So in the first step we take an immutable fast snapshot of the state we care about which happens quickly, then writes can keep going while in the background we serialize the state to disk.
> Imagine all the wonderful things you could build if you never had to serialize data into SQL queries.
This exists in sufficiently mature Actor model[0] implementations, such as Akka Event Sourcing[1], which also addresses:
> But then comes the important part: how do you recover when your process crashes? It turns out that answer is easy, periodically just take a snapshot of everything in RAM.
Intrinsically and without having to create "a new architecture for web development". There are even open source efforts which explore the RAFT protocol using actors here[2] and here[3].
I have built some medium sized systems using Microsoft Orleans (Virtual Actors).
There was no transactional database involved, but everything was ordered and fully transactional.
If you choose say Cosmos DB, MongoDB or DynamoDB as your persistence provider you can even query the persisted state.
When I start a new project, the data structure usually is a "list of items with attributes". For example right now, I am writing a fitness app. The data consists of a list of exercises and each exercise has a title, a description, a video url and some other attributes.
I usually start by putting those items into YAML files in a "data" directory. Actually a custom YAML dialect without the quirks of the original. Each value is a string. No magic type conversions. Creating a new item is just "vim crunches.yaml" and putting the data in. Editing, deleting etc all is just wonderfully easy with this data structure.
Then when the project grows, I usually create a DB schema and move the items into MariaDB or SQLite.
This time, I think I will move the items (exercises) into a JSON column of an SQLite DB. All attributes of an item will be stored in a single JSON field. And then write a little DB explorer which lets me edit JSON fields as YAML. So I keep the convenience of editing human readable data.
Writing the DB explorer should be rather straight forward. A bit of ncurses to browse through tables, select one, browse through rows, insert and delete rows. And for editing a field, it will fire up Vim. And if the field is a JSON field, it converts it to YAML before it sends it to Vim and back to JSON when the user quits Vim.
I'm baffled at the arguments made in this article. This is supposed to be a simpler and faster way to build stateful applications?
The premises are weak and the claims absurd. The author uses overstatement of the difficulties of serialization just to make their weak claim stronger.
> Imagine all the wonderful things you could build if you never had to serialize data into SQL queries.
No transactions, no WAL, no relational schema to keep data design sane, no query planner doing all kinds of optimisations and memory layout things I don't have to think about?
You could say that transactions, for example, would be redundant if there is no external communication between app server and the database. But it is far from the only thing they're useful for. Transactions are a great way of fulfilling important invariants about the data, just like a good strict database schema. You rollback a transaction if an internal error throws. You make sure that transaction data changes get serialised to disk all at once. You remove a possibility that statements from two simultaneous transactions access the same data in a random order (at least if you pick a proper transaction isolation level, which you usually should).
> You also won’t need special architectures to reduce round-trips to your database. In particular, you won’t need any of that Async-IO business, because your threads are no longer IO bound. Retrieving data is just a matter of reading RAM. Suddenly debugging code has become a lot easier too.
Database is far from the only other server I have to communicate with when I'm working on user's HTTP request. As a web developer, I don't think I've worked on a single product in the last 4 years that didn't have some kind of server-server communication for integrations with other tools and social media sites.
> You don’t need crazy concurrency protocols, because most of your concurrency requirements can be satisfied with simple in-memory mutexes and condition variables.
Ah, mutexes. Something that programmers never shot themselves in a foot with. Also, deadlocks don't exist.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. So if you have a line like foo.setBar(2), this will first write a transaction that says we’ve changed the bar field of foo to 2, and then actually set the field to 2. An operation like new Foo() writes a transaction to disk to say that a Foo object was created, and then returns the new object.
A disk write latency is added to every RAM write. It has no performance cost and nobody notices this.
I apologise if this comes off too snarky. Despite all of the above, I really like this idea — and already think of implementing it in a hobby project, just to see how well it really works. I'm still not sure if it's practical, but I love the creative thinking behind this, and a fact that it actually helped them build a business.
I would add that the 'serialization' to a RDBMS-schema cites as a negative is actually a huge positive for most systems. Modeling your data relationally, often in 3NF, usually differs from the in-memory/code objects in all but the most simple ORM class=table projects. Thinking deeply about how to persist data in a way that makes it flexible and useful as application needs change (i.e. the database outlives the applications(s)) has value in itself, not just a pointless cost.
I like being able to draw a hard line between application data structures, often ephemeral and/or optimized for particular tasks -- and the persisted, domain data which has meaning beyond a specific application use case.
186 comments
[ 4.4 ms ] story [ 234 ms ] threadThe difference is that Redis costs extra in infrastructure.
The argument that im fighting agaist is that when someone says its more complex, what they mean is that they dont have experience in doing that. From a business perspective, this is something to consider when hiring from.an average pool, since you point about salary is correct, but the assumption that every single engineer fits this criteria is not correct.
Redis is/was basically just Tcl-typed which are persisted to disk using snapshots (Tcl commands) and append-only Tcl commands, that had a network protocol for non-Tcl applications to talk to
otherwise I get the messaging with edge you the database is the bottleneck
just need a one stop shop to do edge functions + edge db
https://www.microsoft.com/en-us/research/project/orleans-vir...
That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications.
[0] https://www.paulgraham.com/vwfaq.html
Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate.
We also got advanced features like FTS which are useful for desktop and mobile apps.
Today it's a good choice to use RDBMS for desktop apps.
Is there an alternative? I haven't seen a "local filesystem is okay as data storage" software in the 21th century.
Well, there were "options" other than KV stores - MySQL launched a month before Viaweb (but flakey for a good long while.) Oracle was definitely around (but probably $$$$.) mSQL was being used on the web and reasonably popular by 1995 (cheap! cheerful! not terrible!)
(definitely understand making your own in-memory DB in 1995 though)
To be able to read and write you need native os apis, aka read() and write() otherwise how do you do it ?
that was all the rave for prototypes back then.
https://github.com/wting/hackernews/blob/master/news.arc
I think that is just written to disk as something like file41207393 when you click reply.
When the system needs an item it sees if it's cached in memeory and otherwise reads it from disk and I think that is pretty much the whole memory system. Some other stuff like user id that works in the same sort of way.
After reading the link, I don't think that database means the same thing for everyone.
The vwfaq still mentions loading data from disk, and also mention "start up a process to respond to an HTTP request." This suggests that by "database" they meant a separate server dedicated to persist data, and having to communicate with another server to fetch that data.
Obviously, this leaves SQLite out of this definition of database. Also, if you're loading data from disk already, either you're using a database or you're implementing your own ad-hoc persistence layer. Would you still consider you're using a database if you load data from SQLite at app start?
The problem with this sort of mental model is that it ignores the fact that the whole point of a database is to persist and fetch data in a way that is convenient to you without having to bother about low-level details. Storing data in a database does not mean running a postgres instance somewhere and fetching data over the web. If you store all your data in-memory and have a process that saves snapshots to disk using a log-structured data structure... Congratulations, you just developed your own database.
The web has gotten bigger and a lot of these practices simply would not fly today. If I was pushing a live fix on our prod machine with the amount of testing doing it live while on the customer is on the phone entails today, a good portion of you would be questioning my sanity.
I agree, the infrastructure required to make this happen eventually gets quite complicated. But the developer experience is what's super simple. If somebody had to take all our infrastructure and just use it to build their next big app, they can get the simplicity without worrying about the internal plumbing.
Now, with cheap hardware, they’re going back in time to the benefits of clustered, NUMA machines. They’ve improved on it along the way. I did enjoy the article.
Another trick from the past was eliminating TCP/IP stacks from within clusters to knock out their issues. Solutions like Active Messages were a thin layer on top of the hardware. There’s also designs for network routers that have strong consistency built into them. Quite a few things they could do.
If they get big, there’s hardware opportunities. On CPU side, SGI did two things. Their NUMA machines expanded the number of CPU’s and RAM for one system. They also allowed FPGA’s to plug directly into the memory bus to do custom accelerators. Finally, some CompSci papers modified processor ISA’s, networks on a chip, etc to remove or reduce bottlenecks in multithreading. Also, chips like OpenPiton increase core counts (eg 32) with open, customizable cores.
But no, just more lispers.
But I have some bad news: you haven’t built a system without a database, you’ve just built your own database without transactions and weak durability properties.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk.
This is actually not an easy thing to do. If your shutdowns are always clean SIGSTOPs, yes, you can reliably flush writes to disk. But if you get a SIGKILL at the wrong time, or don’t handle an io error correctly, you’re probably going to lose data. (Postgres’ 20-year fsync issue was one of these: https://archive.fosdem.org/2019/schedule/event/postgresql_fs...)
The open secret in database land is that for all we talk about transactional guarantees and durability, the reality is that those properties only start to show up in the very, very, _very_ long tail of edge cases, many of which are easily remedied by some combination of humans getting paged and end users developing workarounds (eg double entry bookkeeping). This is why MySQL’s default isolation level can lose writes: there are usually enough safeguards in any given system that it doesn’t matter.
A lot of what you’re describing as “database issues” problem don’t sound to me like DB issues, so much as latency issues caused by not colocating your service with your DB. By hand-rolling a DB implementation using Raft, you’ve also colocated storage with your service.
> Screenshotbot runs on their CI, so we get API requests 100s of times for every single commit and Pull Request.
I’m sorry, but I don’t think this was as persuasive as you meant it to be. This is the type of workload that, to be snarky about, I could run off my phone[0]
[0]: https://tailscale.com/blog/new-internet
Thanks for the comment! This is handled correctly by Raft/Braft. With Raft, before a transaction is considered committed it must be committed by a majority of nodes. So if the transaction log gets corrupted, it will restore and get the latest transaction logs from the other node.
> I’m sorry, but I don’t think this was as persuasive as you meant it to be.
I wasn't trying to be persuasive about this. :) I was trying to drive home the point that you don't need a massively distributed system to make a useful startup. I think some founders go the opposite direction and try to build something that scales to a billion users before they even get their first user.
I’m now completely lost as to why you believe this was a good idea over using something like MySQL/Postgres/Aurora. As I see it, you’ve added complexity in three different dimensions (novel DB API, novel infra/maintenance, and novel oncall/incident response) with minimal gain in availability and no gain in performance. What am I missing?
(FWIW, I worked on Bigtable/Megastore/Spanner/Firestore in a previous job. I’m pretty familiar with what goes into consensus, although it’s been a few years since I’ve had to debug Paxos.)
> I was trying to drive home the point that you don't need a massively distributed system to make a useful startup. I think some founders go the opposite direction and try to build something that scales to a billion users before they even get their first user.
This reads to me as exactly the opposite: overengineering for a problem that you don’t have.
For exactly the reasons you describe, I would argue the burden of proof is on you to demonstrate why Redis, MySQL, Postgres, SQLite, and other comparable options are insufficient for your use case.
To offer you an example: let’s say your Big Customer decides “hey, let’s split our repo into N micro repos!” and they now want you to create N copies of their instance so they can split things up. As implemented, you’ll now need to implement a ton of custom logic for the necessary data transforms. With Postgres, there’s a really good chance you could do all of that by manipulating the backups with a few lines of SQL.
Isn’t writing «a few Lines of SQL» also custom logic? The difference is just the language.
It is also possible that the custom data store is more easily manipulated with other languages than SQL.
SQL really is great for manipulating data, but not all relational databases are easy to work with.
Yeah. I hope it was clear in my post that the goal was developer productivity, not performance.
The round trip is only an issue on writes, and reads are super fast. At least in my app, this works out great. The writes also parallelize nicely with respect to the round trips, since the underlying Raft library just bundles multiple transactions together. Where it is a bottleneck is if you're writing multiple times sequentially on the same thread.
The solution there is you create a single named transaction that does the multiple writes. Then the only thing that needs to be replicated is that one transaction even though you might be writing multiple fields.
> it’s been a few years since I’ve had to debug Paxos
And this is why I wouldn't have recommended this with Paxos. Raft on the other hand is super easy for anyone to understand.
If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your database”. If your data fits entirely in RAM, then use a ramdisk for the database if you want, and replicate it to permanent storage with standard tools. Now that’s actually simple.
So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar.
That overly-simplified summary aside, I quite like the idea and I think the post does a pretty good job of selling the concept. For a lot of systems it'll scale more than well enough to handle most or all of your business even if you become abnormally successful, and the performance will be absurdly good compared to almost anything else.
https://github.com/rqlite/rqlite
- In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ?
- I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.
If you need to ensure that a given API request (which can contain multiple SQL statements) is atomically processed (all SQL statements succeed or none do) that is supported however [1]. That's why I think of rqlite as closer to the kind of use cases that etcd and Consul support, rather than something like Postgres -- though some people have replaced their use of Postgres with rqlite! [2]
[1] https://rqlite.io/docs/api/api/#transactions
[2] https://www.replicated.com/blog/app-manager-with-rqlite
[1] https://rqlite.io/docs/clustering/general-guidelines/#cluste...
[2] https://rqlite.io/docs/faq/#can-any-node-execute-a-write-req...
The main caveat here is that rqlite is an out-of-process database, which you communicate with over http. That puts it on similar grounds as e.g. postgres, just significantly lighter weight, and somewhat biased in favor of running it locally on every machine that needs the data.
So minimum read latency is likely much lower than postgres, but it's still noticeable when compared to in-process stuff, and you lose other benefits of in-process sqlite, like trivial extensibility.
https://dqlite.io/
I fail to see how this little bit of saving justifies all the complexity for run-of-the-mill web services that fit on one or a few servers as described in the article. The context isn't large scale services where 1ms/request saving translates to $$$, and the proposal doesn't (vertically) scale anyway.
The only thing in my comment that’s not directly based on the article is a handwavy 1ms/request saving estimate, and since they don’t provide any measurement, it’s anyone’s guess.
The guideline specifically advises to do what GP did: Instead of commenting whether or not someone read the article, to tell them that article answers their questions.
But if the blogger learned SQLite, how would they have a topic to blog about?
Also, no benchmarks. It's quite odd that an argument grounded on performance claims does not bother to put out any hard data comparing the output of this project. I'm talking about basic things like how does this contrived custom ad-hoc setup compare with vanilla, out-of-the-box SQLite deployment? Which one performs worse and by how much? How does the performance difference reflect in request times and infrastructure cost? Does it actually pay off to replace the dozen lines of code of on boarding SQLite with a custom, in-development, ad-hoc setup? I mean, I get the weekend personal project vibe of this blog post, but if this is supposed to be a production-minded project then step zero would have been a performance test on the default solution. Where is it?
I probably did a bad job then, because everything in the blog post was meant to be developer productivity claims, not performance claims. (I come from a developer productivity background. I'm decent at performance stuff, but it's not what excites me, since for most companies my size performance is not critical as long as it scales.)
The OP starts out by talking about periodically dumping everything in RAM to disk. I’d say that’s your checkpointing.
Just try and cold-start your database and run a fairly large select twice.
Even if you weren't using erlang, his influence (and in general, ericsson) permeates the telecom industry.
There are libraries available to wrap your stuff with this algorithm, and the benefit is that you write your server like it would run on a single machine, and then when launching it in prod across multiple, everything just works.
A transactional database is simple in Expand and Extract, but adds additional overhead during the Explore phase, because you're focusing on infrastructure issues rather than product. Data reliability isn't critical in the Explore phase either, because you just don't have customers, so you just don't have data.
Having everything in memory with bknr.datastore (without replication) is simple in the Explore phase, but once you get to Expand phase it adds operational overhead to make sure that data is consistent.
But by the time I've reached the Expand phase, I've already proven my product and I've already written a bunch of code. Rewriting it with a transactional database doesn't make sense, and it's easier to just add replication on top of it with Raft.
Unless, of course, your startup is in the business of selling DBMSes.
I think sql db make the most sense during the explore phase and you switch off of them once you know you need an improvement somewhere (like latency or horizontal scalability).
And this comes to the difference between Explore phase and Expand phase.
In the Explore phase, data migration was just running code on the production server via a REPL. Some migrations such as adding/removing fields are just a matter of hot-reloading code in CL, so there weren't a lot of migrations that we had to manually run.
In the Expand phase, once you add replication, this does become hard and we did roll out our own migration framework. But by this point we already have a lot of code built out, so we weren't going to replace it with a transactional database.
Essentially, we optimized for the Explore phase, and "dealt with the consequences" in the Expand phase (but the consequences aren't as bad as you might think).
Then you get used to near-zero latency that in-RAM data gives you, and when it outgrows your RAM, it's a pain in the butt to move it to disk :)
Worse is better until you absolutely need to be less worse, then you'll know for sure. At that point you'll know your pain points and can address them more wisely than building more up front.
For server-based database engines you can still make an argument on shedding network calls. It's dubious, but you can.
What's baffling is that the blogger tries to justify not picking up SQLite claiming it might have features that they don't need, which is absurd and does not justify anything.
The blog post reads like a desperate attempt to start with a poor solution to a fictitions problem and proceed to come up with far-fetched arguments hoping to reject the obvious solution.
Wondering if my thinking is flawed, or if going this—arguably unnecessary—extra mile is part of the product and being successful in the space.
2. Are network requests / other ephemeral things also saved to the snapshot?
Not sure what you mean by ephemeral things. If you mean things like file descriptors, they are not stored. Technically the snapshot is not a simple snapshot of RAM, it snapshots through all the objects in memory that are set up to be part of the datastore. (It's a bit more complicated and flexible than this, but that's the general idea.)
Reading posts like this makes me think the founders/CTO is mixing hobby programming with professional programming.
There's literally no reason to waste time doing all this.
So many lines of pointless, wasted code.
Which is absolutely fine if you are hobby programming but if you are running a business then this approach is wasteful.
>
> And so, if your process crashes and restarts, it first reloads the snapshot, and replays the transaction logs to fully recover the state. (Notice that index changes don’t need to be part of the transaction log. For instance if there’s an index on field bar from Foo, then setBar should just update the index, which will get updated whether it’s read from a snapshot, or from a transaction.)
That’s a database. You even linked to the specific database you’re using [0], which describes itself as:
> […] in-memory database with transactions […]
Am I misunderstanding something?
[0]: https://github.com/bknr-datastore/bknr-datastore
Sound similar to `stop the world Garbage collection` in Java. Does your entire processing comes to halt when you do this? How frequently do you need to take snapshots? Or do you have a way to do this without halting everything
But we aren't really taking the snapshot of RAM, instead we're running some code asking each object to snapshot itself into a stream. If you do this naively, it will block writes on the server until the snapshot is done (reads will continue to work).
But Raft has a protocol for asynchronous snapshots. So in the first step we take an immutable fast snapshot of the state we care about which happens quickly, then writes can keep going while in the background we serialize the state to disk.
This exists in sufficiently mature Actor model[0] implementations, such as Akka Event Sourcing[1], which also addresses:
> But then comes the important part: how do you recover when your process crashes? It turns out that answer is easy, periodically just take a snapshot of everything in RAM.
Intrinsically and without having to create "a new architecture for web development". There are even open source efforts which explore the RAFT protocol using actors here[2] and here[3].
0 - https://en.wikipedia.org/wiki/History_of_the_Actor_model
1 - https://doc.akka.io/docs/akka/current/typed/persistence.html
2 - https://github.com/Michael-Dratch/RAFT_Implementation
3 - https://github.com/invkrh/akka-raft
If you choose say Cosmos DB, MongoDB or DynamoDB as your persistence provider you can even query the persisted state.
https://learn.microsoft.com/en-us/dotnet/orleans/grains/grai...
https://learn.microsoft.com/en-us/dotnet/orleans/grains/tran...
https://learn.microsoft.com/en-us/dotnet/orleans/grains/even...
I usually start by putting those items into YAML files in a "data" directory. Actually a custom YAML dialect without the quirks of the original. Each value is a string. No magic type conversions. Creating a new item is just "vim crunches.yaml" and putting the data in. Editing, deleting etc all is just wonderfully easy with this data structure.
Then when the project grows, I usually create a DB schema and move the items into MariaDB or SQLite.
This time, I think I will move the items (exercises) into a JSON column of an SQLite DB. All attributes of an item will be stored in a single JSON field. And then write a little DB explorer which lets me edit JSON fields as YAML. So I keep the convenience of editing human readable data.
Writing the DB explorer should be rather straight forward. A bit of ncurses to browse through tables, select one, browse through rows, insert and delete rows. And for editing a field, it will fire up Vim. And if the field is a JSON field, it converts it to YAML before it sends it to Vim and back to JSON when the user quits Vim.
The premises are weak and the claims absurd. The author uses overstatement of the difficulties of serialization just to make their weak claim stronger.
These kind of people usually suck to work with. I'm glad they've found a startup to sink so I don't have to deal with them.
No transactions, no WAL, no relational schema to keep data design sane, no query planner doing all kinds of optimisations and memory layout things I don't have to think about?
You could say that transactions, for example, would be redundant if there is no external communication between app server and the database. But it is far from the only thing they're useful for. Transactions are a great way of fulfilling important invariants about the data, just like a good strict database schema. You rollback a transaction if an internal error throws. You make sure that transaction data changes get serialised to disk all at once. You remove a possibility that statements from two simultaneous transactions access the same data in a random order (at least if you pick a proper transaction isolation level, which you usually should).
> You also won’t need special architectures to reduce round-trips to your database. In particular, you won’t need any of that Async-IO business, because your threads are no longer IO bound. Retrieving data is just a matter of reading RAM. Suddenly debugging code has become a lot easier too.
Database is far from the only other server I have to communicate with when I'm working on user's HTTP request. As a web developer, I don't think I've worked on a single product in the last 4 years that didn't have some kind of server-server communication for integrations with other tools and social media sites.
> You don’t need crazy concurrency protocols, because most of your concurrency requirements can be satisfied with simple in-memory mutexes and condition variables.
Ah, mutexes. Something that programmers never shot themselves in a foot with. Also, deadlocks don't exist.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. So if you have a line like foo.setBar(2), this will first write a transaction that says we’ve changed the bar field of foo to 2, and then actually set the field to 2. An operation like new Foo() writes a transaction to disk to say that a Foo object was created, and then returns the new object.
A disk write latency is added to every RAM write. It has no performance cost and nobody notices this.
I apologise if this comes off too snarky. Despite all of the above, I really like this idea — and already think of implementing it in a hobby project, just to see how well it really works. I'm still not sure if it's practical, but I love the creative thinking behind this, and a fact that it actually helped them build a business.
I like being able to draw a hard line between application data structures, often ephemeral and/or optimized for particular tasks -- and the persisted, domain data which has meaning beyond a specific application use case.
I don't know if they're production grade. I was drawn to Braft because of Baidu's backing.