Orama is definitely a hidden gem, and it's a clever usage for complementary indexing!
Also agreed Triplit's DX is excellent. I'd recommend giving it another look, Triplit's recent 1.0 release has up to 10x performance boost (https://www.triplit.dev/blog/triplit-1.0).
Since your use-case is data in the range of gigabytes, you could consider using duckdb-wasm. However I'm not sure how to best integrate this with collaboration / CRDTs (sqlRooms is also interesting prior art).
I’m glad this post made its way to HN, since it allowed me to reflect on how much progress we’ve made since it was written. The blog post doesn’t support comments, so here goes:
> The DX is the worst by quite a margin.
DX is something that we care a lot about. Like it’s a topic in every single planning and strategy session. We’ve made a ton of progress since this was posted, and we have a lot more on this front on our roadmap.
> Not only does it require Postgres-level integration
If you’re looking to stream data from your Postgres database to clients, I’m not aware of any other way to do this but to integrate with Postgres. So I’m not sure why this is framed in a negative light. PowerSync also only requires a user with SELECT privileges, so it’s not an invasive integration.
> it also needs a HA MongoDB cluster
It’s now possible to use Postgres for intermediary storage instead of MongoDB. I actually recall we gave you a shout out in one of our product updates on this :)
> a lot of arcane yaml configuration
We’ve since published a schema for the yaml config - add this to powersync.yaml:
> It also required us to completely denormalise our Postgres tables, as relations are not properly supported across the sync buckets.
This is ultimately a function of our decision to build a system that scales well. Having said that, we have plans to address this limitation this year.
> we found horrifying problems like off-by-one bugs in their Drizzle ORM integration
This was fixed around the time of the post, and our Drizzle integration is now in beta i.e. fully supported in production environments.
> queries returning data from local db very slowly (100ms+),
It’s hard to say what was causing this without knowing more, but as mentioned elsewhere in the comments here we’ve since added support for OPFS which provides a big performance boost.
> and long initialisation times (45s+ after login to Marco) with the UI view not updating until the sync fully completed.
We’ve added APIs to solve this - if you want to sync everything up front you can use sync priorities to log the user in quickly and sync the rest in the background [1]. We’ve also implemented many optimizations like transport-level compression, a waaaay faster rust sync client, etc with more to come. Also, using sync streams (available in the next few weeks) [2] you can sync data just in time as the user navigates around the app instead of syncing everything up front.
> No real case studies
It’s often quite a slow process to get case studies published with large companies, but you can see logos of customers using PowerSync on our website. But yes.
I’m doing offline-first apps at work and want to emphasize that you’re constraining yourself a lot trying to do this.
As mentioned, everything fast(ish) is using SQLite under the hood. If you don’t already know, SQLite has a limited set of types, and some funky defaults. How are you going to take this loosey-goosey typed data and store it in a backend database when you sync? What about foreign key constraints, etc., can you live without those? Some of the sync solutions don’t support enforcing them on the client.
Also, the SQLite query planner isn’t great in my experience, even when you’re only joining on ids/indexes.
Document databases seem more friendly/natural, but as mentioned indexeddb is slow.
I wish this looked at https://rxdb.info/ more. They have some posts that lead me to believe they have a good grasp on the issues in this space at least
Also, OPFS is a newish thing everyone is using to store SQLite directly instead of wrapping IndexedDB for better performance.
I am continually bewildered how no one ever gives RxDB, which has been around for many years longer than the rest of these tools, any love.
It has so many optimizations and features that the others dont. And is even better when you use the premium addons. I compared it to pretty much everything, and its not even close.
I struggled with this landscape a few years ago when building Mere Medical to manage my own medical records. To be fair, I was aiming for not just offline-first, but offline-only (user data was exclusively stored on device, not in any server). I got surprisingly far with RxDB, but it definitely felt like I was pushing these tools and the web platform to their limit.
There’s just an assumption that these client databases don’t need mature tools and migration strategies as “it’s just a web client, you can always just re-sync with a server”. Few client db felt mature enough to warrant building my entire app on as they’re not the easiet to migrate off of.
I also tried LokiJS which is mentioned in the OP. I even forked (renamed it SylvieJS lol) it to rewrite it in TS and update some of the adapters. I ultimately moved away from it as well. I found an in memory db will struggle past a few hundred mbs which I hit pretty quickly.
No matter what db you use, you’re realistically using indexed db behind the hood. What surprised me was that a query to indexed db can be slower than a network call. Like what.
The punchline of this article is that all the implementations they tried (WatermelonDB, PowerSync, ElectricSQL, Triplit, InstantDB, Convex) are all built on top of IndexedDB.
"The root cause is that all of these offline-first tools for web are essentially hacks. PowerSync itself is WASM SQLite... On top of IndexedDB."
But there's a new web storage API in town, Origin Private File System. https://developer.mozilla.org/en-US/docs/Web/API/File_System... "It provides access to a special kind of file that is highly optimized for performance and offers in-place write access to its content."
OPFS reached Baseline "Newly Available" in March 2023; it will be "Widely Available" in September.
WASM sqlite on OPFS is, finally, not a hack, and is pretty much exactly what the author needed in the first place.
Nice post! I'm building an offline-first collaboration app and went on the route of building a custom sync engine, mainly because the app is open-source and I didn't want to introduce any dependency. I've implemented a simple cursor based sync with Postgres on server and SQLite in client side.
Initially I built only a desktop client, because I didn't like IndexedDB. After the app got into HN, someone recommended to check for OPFS (Origin Private File System).
Now we have a full offline-first app in web using SQLite on top of OPFS. We didn't test it with large scale yet, but so far looks very promising. The good thing is that we use Kysely as an abstraction for performing queries in SQLite which helps us share most of the code across both platforms (electron + web) with some minor abstractions.
Related question to people building local-first - what size of db is too big? I always see examples doing todo lists etc which seems perfect for this. But what about apps with larger databases. When does local-first no longer make sense?
Except in pre-sales, I can’t imagine anyone using an email heavy workflow in 2025.
In my personal life, email is only for one way transactions. Where some company is sending email to me or spam. Even the one newsletter I subscribe to - Stratechery is available as a podcast and an RSS feed.
In my professional life, of course all internal communication happens on Slack (700 employees) and even in consulting, the first thing we do after a deal closes is either invite customers to our Slack or ask to be invited to their platform.
What do you mean by this? I send 5-15 emails a day at a minimum throughout the day and receive just as many directly with another 2-3x as cc in various distribution lists (which I read in full). Add in server notifications, automated reports from data processing scripts, and the generic info@company.com inbox and it's probably close to 100 in a day with ease. Lots of skimming and Ctrl-Q'ing and it's hardly a burden.
The lasting power of email is that it's one of the few federated communication channels that has a global network effect. Email and chat are two different media for different purposes. You have plausible deniability when a single message in a group chat is missed. When an email is sent to the team with a change in procedure you can have some expectations that it will be seen and it also provides a one-one or one-many channel for clarification.
I'm not familiar with how the sales world works but I use email every day with clients, vendors, the team, my boss(es), and many other intra-company relationships. I think you have a lack of imagination in this regard :)
Or you could … just build it directly on indexedDB. That's what we did for our offline support at Fastmail, with just a small wrapper function to make the API promise based: https://www.fastmail.com/blog/offline-architecture/
The performance has been pretty decent, even with multi-gigabyte mailboxes.
I find InstantDB's page confusing: How far is it open-source and self-hostable ? I don't mind you having a sustainable cash flow, but it all seems a bit unclear which parts are fully open-source and self-hostable.
I wrap that in FlatDB, which is an opinionated flat cache for the files with metadata inline, used for very fast searches (searching 150k messages in less than 20ms on my 4 year old phone). This handles a lot of tricky cases, like accidental cache modification, and editing the database in different tabs.
Triplit and Orama are definitely often overlooked hidden gems.
Since the post is already a few months old, it's worth mentioning that the newly released Triplit 1.0 had had a massive performance update (up to 10x). You should definitely reconsider it for larger scale data projects and the team is really highly knowledgable. https://www.triplit.dev/blog/triplit-1.0
These topic and mentioned tools always fail to demonstrate how sync engine works in multiple players modes. And mostly only for backend written in TS/JS, worse it's only for using with 3rd service (which you couldn't put your business logic in here). The first thing i scan in these tools' docs are how they handle write conflict whose conflict is semantic level, not data exchange format level.
Triplit has migrated its data format away from triples since this post was authored, so the memory concerns mentioned are no longer relevant since 1.0 https://www.triplit.dev/blog/triplit-1.0 (source: i'm Triplit author). I don't think triples as a format are inherently bad but for us it did entail more data on disk, more difficult and slower querying, and more objects in the JS heap ballooning RAM.
Now you're storing all the rows as units, right?
When users create tables in Triplit, do they end up with really different tables or are they more more like a single table of content in json + table_meta_ref?
I would love to know more about architecture and the changes that you have made
So, the final choice is dexie + custom backend? I've researched these open source solution before, I think dexie is the choice that you will never regret.
29 comments
[ 3.1 ms ] story [ 40.7 ms ] threadWe've had great success with Replicache+Orama since this was written. We're keen to give Zero a spin once it's a bit more stable.
Triplit has essentially folded as a "company" and become some sort of open-source initiative instead.
InstantDB has matured massively and is definitely worth a look for anyone starting a new project.
Also agreed Triplit's DX is excellent. I'd recommend giving it another look, Triplit's recent 1.0 release has up to 10x performance boost (https://www.triplit.dev/blog/triplit-1.0).
Since your use-case is data in the range of gigabytes, you could consider using duckdb-wasm. However I'm not sure how to best integrate this with collaboration / CRDTs (sqlRooms is also interesting prior art).
We have hundreds of thousands of entities in Replicache, and index them via Orama. We're able to perform full-text search in single-digit ms.
We persist the Orama index as JSON, so computation only happens once per mutation.
I’m glad this post made its way to HN, since it allowed me to reflect on how much progress we’ve made since it was written. The blog post doesn’t support comments, so here goes:
> The DX is the worst by quite a margin.
DX is something that we care a lot about. Like it’s a topic in every single planning and strategy session. We’ve made a ton of progress since this was posted, and we have a lot more on this front on our roadmap.
> Not only does it require Postgres-level integration
If you’re looking to stream data from your Postgres database to clients, I’m not aware of any other way to do this but to integrate with Postgres. So I’m not sure why this is framed in a negative light. PowerSync also only requires a user with SELECT privileges, so it’s not an invasive integration.
> it also needs a HA MongoDB cluster
It’s now possible to use Postgres for intermediary storage instead of MongoDB. I actually recall we gave you a shout out in one of our product updates on this :)
> a lot of arcane yaml configuration
We’ve since published a schema for the yaml config - add this to powersync.yaml:
# yaml-language-server: $schema=https://unpkg.com/@powersync/service-schema@latest/json-sche...
> It also required us to completely denormalise our Postgres tables, as relations are not properly supported across the sync buckets.
This is ultimately a function of our decision to build a system that scales well. Having said that, we have plans to address this limitation this year.
> we found horrifying problems like off-by-one bugs in their Drizzle ORM integration
This was fixed around the time of the post, and our Drizzle integration is now in beta i.e. fully supported in production environments.
> queries returning data from local db very slowly (100ms+),
It’s hard to say what was causing this without knowing more, but as mentioned elsewhere in the comments here we’ve since added support for OPFS which provides a big performance boost.
> and long initialisation times (45s+ after login to Marco) with the UI view not updating until the sync fully completed.
We’ve added APIs to solve this - if you want to sync everything up front you can use sync priorities to log the user in quickly and sync the rest in the background [1]. We’ve also implemented many optimizations like transport-level compression, a waaaay faster rust sync client, etc with more to come. Also, using sync streams (available in the next few weeks) [2] you can sync data just in time as the user navigates around the app instead of syncing everything up front.
> No real case studies
It’s often quite a slow process to get case studies published with large companies, but you can see logos of customers using PowerSync on our website. But yes.
[1] https://docs.powersync.com/usage/use-case-examples/prioritiz... [2] https://github.com/powersync-ja/powersync-service/discussion...
As mentioned, everything fast(ish) is using SQLite under the hood. If you don’t already know, SQLite has a limited set of types, and some funky defaults. How are you going to take this loosey-goosey typed data and store it in a backend database when you sync? What about foreign key constraints, etc., can you live without those? Some of the sync solutions don’t support enforcing them on the client.
Also, the SQLite query planner isn’t great in my experience, even when you’re only joining on ids/indexes.
Document databases seem more friendly/natural, but as mentioned indexeddb is slow.
I wish this looked at https://rxdb.info/ more. They have some posts that lead me to believe they have a good grasp on the issues in this space at least
Also, OPFS is a newish thing everyone is using to store SQLite directly instead of wrapping IndexedDB for better performance.
It has so many optimizations and features that the others dont. And is even better when you use the premium addons. I compared it to pretty much everything, and its not even close.
There’s just an assumption that these client databases don’t need mature tools and migration strategies as “it’s just a web client, you can always just re-sync with a server”. Few client db felt mature enough to warrant building my entire app on as they’re not the easiet to migrate off of.
I also tried LokiJS which is mentioned in the OP. I even forked (renamed it SylvieJS lol) it to rewrite it in TS and update some of the adapters. I ultimately moved away from it as well. I found an in memory db will struggle past a few hundred mbs which I hit pretty quickly.
No matter what db you use, you’re realistically using indexed db behind the hood. What surprised me was that a query to indexed db can be slower than a network call. Like what.
"The root cause is that all of these offline-first tools for web are essentially hacks. PowerSync itself is WASM SQLite... On top of IndexedDB."
But there's a new web storage API in town, Origin Private File System. https://developer.mozilla.org/en-US/docs/Web/API/File_System... "It provides access to a special kind of file that is highly optimized for performance and offers in-place write access to its content."
OPFS reached Baseline "Newly Available" in March 2023; it will be "Widely Available" in September.
WASM sqlite on OPFS is, finally, not a hack, and is pretty much exactly what the author needed in the first place.
Initially I built only a desktop client, because I didn't like IndexedDB. After the app got into HN, someone recommended to check for OPFS (Origin Private File System).
Now we have a full offline-first app in web using SQLite on top of OPFS. We didn't test it with large scale yet, but so far looks very promising. The good thing is that we use Kysely as an abstraction for performing queries in SQLite which helps us share most of the code across both platforms (electron + web) with some minor abstractions.
You can check the implementation in Github: https://github.com/colanode/colanode
Except in pre-sales, I can’t imagine anyone using an email heavy workflow in 2025.
In my personal life, email is only for one way transactions. Where some company is sending email to me or spam. Even the one newsletter I subscribe to - Stratechery is available as a podcast and an RSS feed.
In my professional life, of course all internal communication happens on Slack (700 employees) and even in consulting, the first thing we do after a deal closes is either invite customers to our Slack or ask to be invited to their platform.
This is a solution in search of a problem.
What do you mean by this? I send 5-15 emails a day at a minimum throughout the day and receive just as many directly with another 2-3x as cc in various distribution lists (which I read in full). Add in server notifications, automated reports from data processing scripts, and the generic info@company.com inbox and it's probably close to 100 in a day with ease. Lots of skimming and Ctrl-Q'ing and it's hardly a burden.
The lasting power of email is that it's one of the few federated communication channels that has a global network effect. Email and chat are two different media for different purposes. You have plausible deniability when a single message in a group chat is missed. When an email is sent to the team with a change in procedure you can have some expectations that it will be seen and it also provides a one-one or one-many channel for clarification.
I'm not familiar with how the sales world works but I use email every day with clients, vendors, the team, my boss(es), and many other intra-company relationships. I think you have a lack of imagination in this regard :)
In my experience, email seems common for customer support.
The performance has been pretty decent, even with multi-gigabyte mailboxes.
* We added some serious typescript types
* We have sorting and ordering on fields
* We added the $like operator
* We added reactive queries on the backend
We're on a mission to make the best DX possible for building apps. We take your feedback seriously, and ship as quick as we can.
As the author mentions in the comments here:
> InstantDB has matured massively and is definitely worth a look for anyone starting a new project.
If you get a chance to try us out, we'd love feedback :)
I wrap that in FlatDB, which is an opinionated flat cache for the files with metadata inline, used for very fast searches (searching 150k messages in less than 20ms on my 4 year old phone). This handles a lot of tricky cases, like accidental cache modification, and editing the database in different tabs.
There's even a sync algorithm that synchronizes notes across devices. https://github.com/kasrasadeghi/pipeline-js/blob/main/assets...
Since the post is already a few months old, it's worth mentioning that the newly released Triplit 1.0 had had a massive performance update (up to 10x). You should definitely reconsider it for larger scale data projects and the team is really highly knowledgable. https://www.triplit.dev/blog/triplit-1.0
P.S. I am the author
fyi https://dexie.org/
On mobile and desktop, IndexedDB does not exist, and SQLite is available first-class.