Great link. I've always been drawn to sqlite3 just from a simplicity and operational point of view. And with tools like "make it easy to replcate" Litestream and "make it easy to use" sqlite-utils, it just becomes easier.
And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at my last company -- and it was a mistake, when a simple DB would have sufficed.
I didn't know about Tansu and probably would not use it for anything too serious (yet!). Bus as a firm believer of event sourcing and change of paradigm that Kafka brings this is certainly interesting for small projects.
Quite cool. 7000 records per second is usable for a lot of projects.
One note on the backup/migrate, I think you need a shared lock on the database before you copy the database. If you dont, the database can corrupt. SQLite docs have other recommendations too:
Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compatible, but trivial to run, system like this is very intriguing!
Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience).
So I'd be interested how (backward-)compatible they are.
This SQLite obsession is getting quite ridiculous. Now they put it in "the Cloud." What a shitshow. I wonder whether they know what SQLite is for... when Cloudflare did it, well, it made sense at least. This new generation of SQLite caro-culting is beyond anything I've ever seen.
Peter (the author) is a really, really cool guy. We recorded a 3hr 30m podcast[0] with him a month ago. For anyone interested in the Kafka space, performance optimization in Rust and the general "why yet another Kafka", I'd shamelessly recommend the video:
I love sqlite backed system, one less component to worry about. But when using Tansu with sqlite storage, what are my options for horizonal scaling and keeping Tansu HA?
Also, are there any benchmark on how Tansu with S3 storage would perform in comparison to Kafka or something like WarpStream?
11 comments
[ 4.4 ms ] story [ 30.6 ms ] threadAnd one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at my last company -- and it was a mistake, when a simple DB would have sufficed.
https://github.com/simonw/sqlite-utils https://litestream.io/
One note on the backup/migrate, I think you need a shared lock on the database before you copy the database. If you dont, the database can corrupt. SQLite docs have other recommendations too:
https://sqlite.org/backup.html
same for redis, kafka, ...
> kafka compatible
Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience).
So I'd be interested how (backward-)compatible they are.
[0] - https://www.youtube.com/watch?v=pJQ7hcsI1Dw
Also, are there any benchmark on how Tansu with S3 storage would perform in comparison to Kafka or something like WarpStream?