While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.
There are plenty of use-cases where the risk of losing data to gain performance is perfectly acceptable.
What is important is being clear about what trade-offs they make.
As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings.
All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting.
There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example.
This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.
That's true, but it's got to be a trade-off and a balance of risk, because "Performance over everything" makes no sense at all: the most performant thing to do is to store no data and return no data.
And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results.
So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.
And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is on disk. If caches don't exist then you pull from disk (slow) and write to the caches.
this is actually something we are playing with :) we have a couple of drafts in the pipe but you are welcome to bring some wisdom over to LokiJS - it's free for all to use and for all to contribute, even if it's just on a theoretical level. Cheers
Honestly the wisdom I have has just been playing around with benchmark.js on V8 and noticing the Chrome team doesn't do as they claim (although latest Chrome Canary we saw major improvements), while FireFox is a different beast. You've probably done about just as much of this as me.
What we found was that every function call in javascript significantly decreased performance (even if it did nothing) at a quazi logarithmic rate. So on a theoretical level if you're able to reuse function results that are deterministic you can save a ton of overhead. However IDK how micro-optimizing you want to get as it can sacrifice code legibility.
Would love to chat more if you have time, I'll see if I can find your email and send you a message. Mine is (no spaces): a q u i v a @ google's email provider.
Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture.
The Typeform LITE mobile application is built by using CouchDB and PouchDB which means when the application is online, it replicates the users db locally so the performance is great but the even bigger benefit is that when the application is offline, the user can still do the majority of the operations which would be synced once the application goes online again.
So, probably no-one wants to replicate the whole database locally but there is a strong benefit to replicate parts of it for performance and offline-capabilities.
Disclaimer: I was one of two developers on the mobile app mentioned above.
You can filter what you are syncing or create a db per user and declare interactions with foreign data in their own instance, which will be executed according to permissions in the backend.
What I actually want is MeteorJS with neither MongoDB or the JavaScript backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open source compatibility...
I too haven't seen anything on Loki's sync ability. Obviously would be a very nice feature but difficult to do right.
Pouch, Firebase, and GUN[1] (disclaimer, I work there) are examples. If you bake permissions into the system itself, you get monoliths like Meteor. But you can still achieve granular permissions by rejecting/accepting network requests (like with ExpressJS or Hapi, etc.), which gives you the balance of using whatever backend you like.
syncing isn't as straight forward mainly because not all DBs have replication protocols like CouchDB does. However, LokiJS provides a Changes API which serializes changes from a set point in time so you can easily process those.
What I find fascinating about a tool like this is how simple it is. An in memory store is not very important to me and what I develop. And yet, Im very happy this exists so that it sets a standard for other work in the future.
V8 has some quirky memory mechanics. Object pointers, and similar, are subject to a vague hard limit, loosely defined by `--max-old-space-size` and other related flags, while the objects themselves can extend beyond it. For example, if you had 1 object that contained 15gb text, that would likely be OK, however if you had 1 billion objects, storing 1 character each, that would likely lead to a crash - even if the process memory is below limits. Relatedly the shape and nature of the data matters (nesting, types, etc.), limiting the ability to use the full memory available in various ways. I imagine a lot of it has to do with GC, but I don't think that's the full story.
Have you noticed such quirks building loki? Do you imagine being able to make full use of, say, a 64gb ram server? Is this something being taken into account and optimized in the project somehow? @joeminichino
I used this on a internal project for a little while, it's easy to use, and the promises integration is useful, however at high volumes it did very poorly.
I was storing time series data.
I ended up moving to redis which has worked very well as long as I make sure it has enough memory for the date range I am keeping.
I hope this project keeps at it and gets a handle on performance under heavy load.
hey @ldehaan i'd love to hear more about the shortcomings of Loki, i never had a real-life need for really high volumes so I would greatly appreciate some insights in order to improve it.
After 5min of introduction video I had to stop. Guy explains why he started loki. Hi didn't want to use sqlite in cordova app, so he created loki. Did he heard about localStorage or indexeddb? Probably not.
30 comments
[ 0.73 ms ] story [ 74.3 ms ] threadWhat is important is being clear about what trade-offs they make.
As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings.
All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting.
There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example.
This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.
And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results.
So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.
And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is on disk. If caches don't exist then you pull from disk (slow) and write to the caches.
Well, no shit?
Honestly the wisdom I have has just been playing around with benchmark.js on V8 and noticing the Chrome team doesn't do as they claim (although latest Chrome Canary we saw major improvements), while FireFox is a different beast. You've probably done about just as much of this as me.
What we found was that every function call in javascript significantly decreased performance (even if it did nothing) at a quazi logarithmic rate. So on a theoretical level if you're able to reuse function results that are deterministic you can save a ton of overhead. However IDK how micro-optimizing you want to get as it can sacrifice code legibility.
Would love to chat more if you have time, I'll see if I can find your email and send you a message. Mine is (no spaces): a q u i v a @ google's email provider.
Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture.
[1]: https://pouchdb.com/
Who wants to replicate their "whole" database locally?
So, probably no-one wants to replicate the whole database locally but there is a strong benefit to replicate parts of it for performance and offline-capabilities.
Disclaimer: I was one of two developers on the mobile app mentioned above.
What I actually want is MeteorJS with neither MongoDB or the JavaScript backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open source compatibility...
Pouch, Firebase, and GUN[1] (disclaimer, I work there) are examples. If you bake permissions into the system itself, you get monoliths like Meteor. But you can still achieve granular permissions by rejecting/accepting network requests (like with ExpressJS or Hapi, etc.), which gives you the balance of using whatever backend you like.
[1]: https://github.com/amark/gun
Have you noticed such quirks building loki? Do you imagine being able to make full use of, say, a 64gb ram server? Is this something being taken into account and optimized in the project somehow? @joeminichino
I was storing time series data.
I ended up moving to redis which has worked very well as long as I make sure it has enough memory for the date range I am keeping.
I hope this project keeps at it and gets a handle on performance under heavy load.