> But the market has spoken! Users prefer Litestream. And honestly, we get it: Litestream is easier to run and to reason about. So we’ve shifted our focus back to it.
Really love litestream. Easy to use and never crashes on me. I still recommend using it as a systemd unit service.
I'm not only using it as a backup tool but also to mirror databases. Looking forward to their read-replica feature.
Maybe I misunderstand what this is, but why would I use this and not MySQL, Postgres, or any other proper database? Seems like a hack to get SQLite to do what those do by design.
I once was responsible for migrating a legacy business app to Azure, and the app had a local MSSQL server co-running with the app (the same pattern that Litestream is using).
As have been mentioned below, the app had been developed assuming the local access (and thus <1ms latency), so it had a ton of N+1 everywhere.
This made it almost impossible to migrate/transition to another configuration.
So, if this style of app hosting doesn't take off and you're at all worried about this being a dead end storage once you reach a certain scale, I'd recommend not doing this, otherwise your options will be very limited.
Then again - I bet you could get very very far on a single box, so maybe it'd be a non factor! :)
Very excited to see Fly restart development on Litestream after a 2ish year freeze!
I love Litestream and use it in every app I build now.
They advertise it as costing "pennies per day," but it's even less expensive than that. It obviously varies depending on how much storage you need, but I had a real app in production, and Litestream replication to S3 only cost me 2-3 cents ($0.02-$0.03) per month.[0]
What I'd like to see is a system where a single-writer SQLite database is replicated to object storage so that you can spin up really cheap read replicas. Is anyone working on something like that?
Such a system would also require a side channel propagating WAL updates (over Kafka or similar) to replicas, so that the running replicas can update themselves incrementally and stay fresh without loading anything from object storage.
Interesting information that they chose modernc.org/sqlite over mattn/go-sqlite3 as a Quality-of-Life improvement. Going forward I guess I'll do the same for new projects.
The NATS Jetstream use case is something I'm curious about.
We have an in house application installed on a remote fleet with spotty internet access. Because of the spotty internet we have a lot of trouble setting up a reliable system for getting data home.
Could how does litestream handle backing up through a spotty connection and can we consolidate the backups into a central db an query against it?
i am not a big fly.io fan per se, but their blog posts, especially in this sort of sphere, is delightful, and i absolutely commend them for this. fly has a fascinating combination of “dev-forward” and “we made this complicated because it shits us” that continues to confound me, but i must confess i am a big fan of this project, and a number of other projects (e.g. svelte) that they seem to genuinely treat in an arms-length, “let a thousand flowers bloom” sort of way.
I look forward to trying this out. Any benchmarks or demos on how long it actually takes to restore? I ended up cooking my own boring S3 backup because previously litestream took 20 minutes to restore something like 1000 rows. It felt extremely unoptimized. How long does restoration take today?
FWIW. We saw this and I started experimenting. A simple case of copying our production database locally a few times. About 1 in 4 times the local database was corrupted after running it. I also experienced very strange command line parsing bugs and different behaviour depending on the name of the local database file.
So I would treat sqlite3_rsync as more of a demo than a stable product right now.
Litestream provides near-real-time offsite replication and point in time recovery, which sqlite3_rsync won't do on its own. You could probably build a litestream-like product based on sqlite3_rsync but it probably won't be as fast or as efficient on storage.
The DX for deploying SQLite apps to Fly.io is rough. I'm a few hours into trying to get a production Rails app booting, but running into issues getting the database to initialize, migrate, and become writable. The root of my problem was the eager loading of a gem I wrote, but there were several layers of runners above it that made it hard to diagnose.
I wish they'd put a bit more effort into the DX here, but it probably doesn't make much sense from a biz PoV since big customers aren't going to be running these kinds workloads.
Curious if anybody here is deploying SQLite apps to production and what host they're using?
> In Litestream, we’re solving the problem a different way. Modern object stores like S3 and Tigris solve this problem for us: they now offer conditional write support. With conditional writes, we can implement a time-based lease. We get essentially the same constraint Consul gave us, but without having to think about it or set up a dependency.
Reading this blog post though, I couldn't see any reference to this. Is this supported in Litestream v0.5.0, or will it be for a future release?
FUSE is such a cool idea but every application always suffers performance problems when stressed. I tried LiteFS and it worked, but litestream was awesome.
Question about using Litestream - I have an app running, and now need to upgrade it to a newer version. Currently I build the new version in a different directory, stop the old app, copy/migrate the database from the old version to the new version (in the new directory). Then finally I delete the old directory, and rename the new directory to be the production one.
Will Litestream freak out about the database being replaced underneath it?
Will I still be able to restore old versions of the DB?
24 comments
[ 4.6 ms ] story [ 47.0 ms ] thread> But the market has spoken! Users prefer Litestream. And honestly, we get it: Litestream is easier to run and to reason about. So we’ve shifted our focus back to it.
https://fly.io/blog/litestream-revamped/ https://news.ycombinator.com/item?id=44045292
I once was responsible for migrating a legacy business app to Azure, and the app had a local MSSQL server co-running with the app (the same pattern that Litestream is using).
As have been mentioned below, the app had been developed assuming the local access (and thus <1ms latency), so it had a ton of N+1 everywhere.
This made it almost impossible to migrate/transition to another configuration.
So, if this style of app hosting doesn't take off and you're at all worried about this being a dead end storage once you reach a certain scale, I'd recommend not doing this, otherwise your options will be very limited.
Then again - I bet you could get very very far on a single box, so maybe it'd be a non factor! :)
I love Litestream and use it in every app I build now.
They advertise it as costing "pennies per day," but it's even less expensive than that. It obviously varies depending on how much storage you need, but I had a real app in production, and Litestream replication to S3 only cost me 2-3 cents ($0.02-$0.03) per month.[0]
[0] https://mtlynch.io/litestream/#using-logpaste-in-production
Such a system would also require a side channel propagating WAL updates (over Kafka or similar) to replicas, so that the running replicas can update themselves incrementally and stay fresh without loading anything from object storage.
https://developers.cloudflare.com/d1/best-practices/read-rep...
The NATS Jetstream use case is something I'm curious about.
Cheers and keep up the great work on Litestream.
Could how does litestream handle backing up through a spotty connection and can we consolidate the backups into a central db an query against it?
So far I’ve stuck with the SFTP solution, since I don’t use any of the cloud object storage services that are hardcoded into the tool.[^2]
Big thanks to the developers.
[^1]: https://github.com/benbjohnson/litestream/pull/731
[^2]: https://litestream.io/guides/#replica-guides
So I would treat sqlite3_rsync as more of a demo than a stable product right now.
Litestream provides near-real-time offsite replication and point in time recovery, which sqlite3_rsync won't do on its own. You could probably build a litestream-like product based on sqlite3_rsync but it probably won't be as fast or as efficient on storage.
I wish they'd put a bit more effort into the DX here, but it probably doesn't make much sense from a biz PoV since big customers aren't going to be running these kinds workloads.
Curious if anybody here is deploying SQLite apps to production and what host they're using?
> In Litestream, we’re solving the problem a different way. Modern object stores like S3 and Tigris solve this problem for us: they now offer conditional write support. With conditional writes, we can implement a time-based lease. We get essentially the same constraint Consul gave us, but without having to think about it or set up a dependency.
Reading this blog post though, I couldn't see any reference to this. Is this supported in Litestream v0.5.0, or will it be for a future release?
Will Litestream freak out about the database being replaced underneath it?
Will I still be able to restore old versions of the DB?