Can't use it locally (yet?) but it's definitely an interesting move in the space. For my personal projects lately I've been defaulting to sqlite in dev, and having a database wrapper layer to use something else in prod.
I’m not in a rush to use a reimplementation of SQLite — particularly from startup bros that had a very public, one-sided, and unpleasant fight with SQLite over their contribution model.
D. Richard Hipp is a genuinely fantastic human being, and SQLite is a project developed at literally the level of planetary infrastructure given how broadly and everywhere it appears.
Forking his project while using the name to garner recognition and attention is poor form, and it is difficult to have faith in the results.
Kind of cool to see work on this. I do hope that the final db file result is still binary compatible with SQLite 3 in whatever direction Turso moves towards though... Rust or not.
I've been advocating with several projects over recent years to get SQLite3 as an archive/export/interchange format for data. Need to archive 2019 data from the database, dump it into a SQLite db with roughly the same schema... Need to pass multiple CSVs worth of data dumps, use a SQLite file instead.
As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection.
The single-writer limitation in SQLite is per-database, not per-connection. You can shard your SQLite tables into multiple database files and query across all of them from a single connection.
I agree that "the single-writer limitation isn't just a theoretical concern", but it's also solvable without forking SQLite. ulimit's the limit! If your goal is resource maximization of a given computer, though, Postgres is likely a better fit.
Even in the “low-performing” zone of the single threaded SQLite and Turso, we are still talking about 50k rows per second and 1000 microseconds, aka 1 ms. It is insanely fast. 1ms is just the round-trip for my Postgres on RDS. It is amazing that SQLite is so awesome. I understand it is not for every use-case, but still awesome
While I understand that you could do this, I genuinely don't understand why.
SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up.
SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton for that, but the idea that your application would have so many threads needing to write concurrently that it would be slowed down by a file lock... just doesn't make sense to me.
What applications need this, and why wouldn't they use Postgres/MySQL/etc. if they need such a high level of concurrent performance? This feels like trying to adapt a small sports car to tow a semi-trailer. It doesn't seem like it's what it's meant for.
I've been thinking Turso in the browser would be some eventual addition, is something the Rust rewrite would eventually unlock. And that would require considerable extra backing to make go. But here it is now!
How does the mvcc / row versioning changes that accumulate get shared among multiple processes? Is it another file like the WAL file? Or is it just local memory as part of the client? I guess with everything having to commit or entirely roll back at the end, it doesn't need to be shared.
How does this compare with hctree then? It sounds similar, both using an MVCC model.
Really cool though, love the technical details and would be interested to see further. I feel like there is certainly a space for sqlite+fancier features. Though it would break backwards compatibility, I've always wondered what kind of performance gains could be had by using a different varint implementation. And enforcing or always using "strict" for smarter column layouts.
Also would be really cool to have indexed views.
And columnstore seems like a logical direction as well.
11 comments
[ 0.96 ms ] story [ 35.7 ms ] threadD. Richard Hipp is a genuinely fantastic human being, and SQLite is a project developed at literally the level of planetary infrastructure given how broadly and everywhere it appears.
Forking his project while using the name to garner recognition and attention is poor form, and it is difficult to have faith in the results.
I've been advocating with several projects over recent years to get SQLite3 as an archive/export/interchange format for data. Need to archive 2019 data from the database, dump it into a SQLite db with roughly the same schema... Need to pass multiple CSVs worth of data dumps, use a SQLite file instead.
As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection.
I agree that "the single-writer limitation isn't just a theoretical concern", but it's also solvable without forking SQLite. ulimit's the limit! If your goal is resource maximization of a given computer, though, Postgres is likely a better fit.
Though this stuff moves slowly (that announcement was almost 3 years ago!), so I'm glad to see Turso giving us options today.
SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up.
SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton for that, but the idea that your application would have so many threads needing to write concurrently that it would be slowed down by a file lock... just doesn't make sense to me.
What applications need this, and why wouldn't they use Postgres/MySQL/etc. if they need such a high level of concurrent performance? This feels like trying to adapt a small sports car to tow a semi-trailer. It doesn't seem like it's what it's meant for.
I've been thinking Turso in the browser would be some eventual addition, is something the Rust rewrite would eventually unlock. And that would require considerable extra backing to make go. But here it is now!
How does this compare with hctree then? It sounds similar, both using an MVCC model.
Really cool though, love the technical details and would be interested to see further. I feel like there is certainly a space for sqlite+fancier features. Though it would break backwards compatibility, I've always wondered what kind of performance gains could be had by using a different varint implementation. And enforcing or always using "strict" for smarter column layouts.
Also would be really cool to have indexed views.
And columnstore seems like a logical direction as well.