I discovered Litestream here on HN after benbjohnson shared it here a few months back.[0] I wasn't interested at first because I don't ever use SQLite, but it stuck with me. I eventually realized it's a great way to achieve vendor flexibility in hosting apps, so I used Litestream to build a pastebin-style service for my app and wanted to share what I learned.
Happy to answer any questions or hear any feedback about this post.
Just wanted to say thank you for writing this Michael. You've been putting lots of great content on your blog! Really enjoying the real world examples you include.
Question from a newbie to cloud stuff: what is the performance like for a database server that's separated from your host like that?
In terms of latency alone it sounds like it risks adding tens or even hundreds of milliseconds to even a fairly trivial request: out of my datacenter, across the wire, into theirs, grab the data, and then reverse the path. And perhaps a charge for data going out each way.
Am I wrong about that? Am I overestimating the travel time, or perhaps they end up being located in the same datacenter so it's all high speed?
It's typical to see at least millisecond latency for a database server that's located in the same data center (e.g. hosting your own Postgres) and tens or hundreds of milliseconds for cloud database services that run outside your data center.
The OP's approach of SQLite & Litestream removes that latency entirely since the database is colocated with the application. Litestream will stream backups to an external service (e.g. S3) and that can have tens or hundreds of milliseconds of latency but that won't affect the application's performance since it is asynchronous replication.
what is the performance like for a database server that's separated from your host like that?
It's not separated. As this diagram shows - https://mtlynch.io/litestream/diagram.jpg - the SQLite database is on the same server as the web application. Additionally, OP's setup seems to use Litestream to backup the SQLite database to some cloud storage.
Yep, you do have to download the DB on every deploy. It's negligible for my deploys because my database is a few MB, but if it got to be in the GB range, I'd have to find some way around that.
> It’s also important to note that Litestream can’t resolve conflicts between multiple database writes, so each database can have only one application server with write access.
It’s cool, don’t get me wrong, but given the above constraint isn’t it just:
Litestream is doing more magic than it seems, so it's much better than a naive while loop.
Litestream is only streaming the changes in the database, whereas the naive implementation would have to upload the entire database every second.
Also, SQLite has a write-ahead log, so you're not guaranteed that all the data is always in the main .db file. Litestream watches the write-ahead log files as well and immediately streams those changes to cloud storage. A system that only watches the main .db file would have long time windows where data loss is possible.
dqlite and rqlite are both great projects but target different use cases. Those run Raft consensus which requires at least 3 nodes for HA whereas Litestream is meant to run on a single node and trades off with lower durability guarantees.
Litestream flushes out changes to S3 on an interval. By default, it’s every 10 seconds although you could reasonably drop that down to 1 second. You could go lower if you’re replicating to NFS.
That interval determines your window for data loss in the event of a catastrophic failure (e.g. disk failure). I’ve run multiple DigitalOcean machines for years and have never had a catastrophic failure so they are rare events but they can happen.
Dqlite and rqlite run every write through a distributed consensus algorithm which guarantees that your writes are persisted across at least a majority of nodes in your cluster before returning a success. In that situation, you’d need a catastrophic failure on a majority of nodes to lose data, however, there is generally a large performance trade off when running distributed consensus.
Is it possible to use dqlite/rqlite for high availability in the same datacenter, while shipping logs using litestream to a different data center for diaster recovery? Or are they incompatible?
Yes, there is no reason why that wouldn't work. rqlite supports on-disk mode so you could run litestream alongside an rqlite node and backup the underlying SQLite database to your favourite cloud provider using litestream.
The only downside is that rqlite performance is very sensitive to the number of writes to disk, and that's why rqlite uses an in-memory SQLite database by default, and lets the Raft log persist to disk. In exchange you can be sure that your writes are persisted to disk when the API acks your request. Perhaps if rqlite used a RAM-based filesystem for everything, and you combined it with litestream you could get much higher-performance from rqlite, with backup (with only a tiny window for data loss) if you use litestream. It's not entirely trivial, however, due the difference in data consistency models. For example, which SQLite database should be updated if the Leader in the cluster changes? It gets complicated.
Ben has written a great program here. I wish I had his ideas! It's got me thinking. :-)
Misleading title. First thought that Litestream had deleted the database or that the service was discontinued for the price.
Better would be "How Litestream replaces a database server for 0.03ct/month" or "Litestream eliminates the need for a Database Server for 0.03ct/month".
Cool little project. If lowest cost is the goal, I am wondering if you could do it too by not having neither sqlite and litestream, and store each log as a separate object on S3.
25 comments
[ 3.4 ms ] story [ 69.1 ms ] threadI discovered Litestream here on HN after benbjohnson shared it here a few months back.[0] I wasn't interested at first because I don't ever use SQLite, but it stuck with me. I eventually realized it's a great way to achieve vendor flexibility in hosting apps, so I used Litestream to build a pastebin-style service for my app and wanted to share what I learned.
Happy to answer any questions or hear any feedback about this post.
[0] https://news.ycombinator.com/item?id=26103776
In terms of latency alone it sounds like it risks adding tens or even hundreds of milliseconds to even a fairly trivial request: out of my datacenter, across the wire, into theirs, grab the data, and then reverse the path. And perhaps a charge for data going out each way.
Am I wrong about that? Am I overestimating the travel time, or perhaps they end up being located in the same datacenter so it's all high speed?
The OP's approach of SQLite & Litestream removes that latency entirely since the database is colocated with the application. Litestream will stream backups to an external service (e.g. S3) and that can have tens or hundreds of milliseconds of latency but that won't affect the application's performance since it is asynchronous replication.
Yep, you do have to download the DB on every deploy. It's negligible for my deploys because my database is a few MB, but if it got to be in the GB range, I'd have to find some way around that.
It’s cool, don’t get me wrong, but given the above constraint isn’t it just:
with extra bells and whistles?Litestream is doing more magic than it seems, so it's much better than a naive while loop.
Litestream is only streaming the changes in the database, whereas the naive implementation would have to upload the entire database every second.
Also, SQLite has a write-ahead log, so you're not guaranteed that all the data is always in the main .db file. Litestream watches the write-ahead log files as well and immediately streams those changes to cloud storage. A system that only watches the main .db file would have long time windows where data loss is possible.
Would a multi tenant SaaS with databases on the much smaller size compared to what you all talk about do well to use this, with proper backups?
That interval determines your window for data loss in the event of a catastrophic failure (e.g. disk failure). I’ve run multiple DigitalOcean machines for years and have never had a catastrophic failure so they are rare events but they can happen.
Dqlite and rqlite run every write through a distributed consensus algorithm which guarantees that your writes are persisted across at least a majority of nodes in your cluster before returning a success. In that situation, you’d need a catastrophic failure on a majority of nodes to lose data, however, there is generally a large performance trade off when running distributed consensus.
p.s. Thanks for litestream!
Yes, there is no reason why that wouldn't work. rqlite supports on-disk mode so you could run litestream alongside an rqlite node and backup the underlying SQLite database to your favourite cloud provider using litestream.
The only downside is that rqlite performance is very sensitive to the number of writes to disk, and that's why rqlite uses an in-memory SQLite database by default, and lets the Raft log persist to disk. In exchange you can be sure that your writes are persisted to disk when the API acks your request. Perhaps if rqlite used a RAM-based filesystem for everything, and you combined it with litestream you could get much higher-performance from rqlite, with backup (with only a tiny window for data loss) if you use litestream. It's not entirely trivial, however, due the difference in data consistency models. For example, which SQLite database should be updated if the Leader in the cluster changes? It gets complicated.
Ben has written a great program here. I wish I had his ideas! It's got me thinking. :-)
isn't this wrong? The S3 may not be the remote DB, but it is sorta is, in this case.
The major thing is the app doesn't have to be coded with s3's latency in mind when it accesses the db.