Show HN: Storex – A modular and portable database abstraction for JavaScript
We'd like to show you Storex, a database abstraction layer that allows you to move your database interaction code between client- and server-side. By surrounding it with modular, recombinable packages for common problems such as schema migrations, it's possible to re-use logic touching the database across a wide variety of databases, allowing you to develop your code entirely in-browser in your daily development workflow, and then move your database to PostgreSQL/MySQL back-end once you're ready. Storex was designed modularly enough to easily be adapted to mBaaS like Firebase, which is coming up soon. Right now it's being used in WorldBrain's Memex (worldbrain.io) to provide you with a client-side full-text searchable database of every page you've seen online and your annotations on them (up to 5GB of data for some of our users.) We'd appreciate your thoughts very much, and don't hesitate to get involved! More info: https://medium.com/worldbrain/storex-a-modular-and-portable-...
21 comments
[ 3.5 ms ] story [ 18.2 ms ] threadThis could absolutely be my own misunderstanding, but you mention that it's explicitly not an ORM, but reading through the docs, it seems like the collections and querying aspects of this project are just that. It actually seems like most of what this project's scope seems to be is to be used as an ORM (which I personally am fine with). Can you elaborate on that at all?
Sharing database abstractions between the client and the server might make sense for some applications, but I worry it might often be a code smell. I used to believe in the dream of sharing models between the client and server, but that illusion was shattered on first contact with reality. With that said, I'm still open to having my mind changed. I think meteor [0] set out with a similar goal, if not more ambitious, and it doesn't appear to have done very well.
The limitations are pretty severe, since you're basically targeting the lowest common denominator. You cannot reduce the inherent complexity of the problem which these databases set out to solve without sacrificing functionality. My thoughts on this have been all over the place, but lately I've been leaning towards adopting fewer abstraction layers. At some point you'll be forced to actually understand each underlying system anyway, and by designing your data models with a specific database in mind you can leverage all of its features to maximum effect in order to achieve optimal performance.
It might be useful to review Django's ORM design for ideas and inspiration, as they also implement the data mapper pattern. You should consider copying Django's migration commands [1] as well.
The following refers mostly to server databases, but some of it can apply to client databases as well. You probably shouldn't be running migrations automatically, especially in production. We're all human and mistakes happen, but there are steps you can take to reduce their potential impact, if not eliminate them entirely within certain processes. Running migrations outside of peak hours reduces the chance you'll bump into performance problems and that it'll affect users. Always create a snapshot or backup before running migrations, data loss is never acceptable! When a user trusts you with their data it's your responsibility to handle it with immense care.
This tool doesn't help with the more challenging problems related to schema management. Zero-downtime migrations are the industry standard. There's a large set of common migration operations which require a full table lock unless they're written with deliberate care so as to avoid those pitfalls. Sever-side migrations must always be done in lock-step, with fully backwards-compatible code capable of handling both versions during a transition. Client-side applications are usually expected to remain largely fully compatible with both backwards and future changes.
[0] https://www.meteor.com/
[1] https://docs.djangoproject.com/en/2.1/topics/migrations/
The first idea is to have the common denominator implemented, and then being able to access or translate things to lower-level operations. Because even though there may be very complex queries happening, a lot of queries are also quite simple, especially when you're starting out. The goal therefore is to minimize, not completely eliminate database-specific code. And even though best practices for data modeling may differ between databases, in the end you want to express and work on relationships between data, whether in SQL, MongoDB, Firebase or Neo4j. If you express your intentions correctly, your database should provide you with options to translate it to common best practices. (Like the decision to embed objects in MongoDB, or do manual cross-relationship queries for example.)
About schema management, this is the first tool I've seen that actually does begin to support you with the more complex issues of migrations (although it's definitely not finished.) Most framework come with a package deal of how you should run your migrations, meaning from their codebase. Storex generates an intermediate structure of what it's going to do during a migration, which you can then decide what to do with (generate an SQL script, run it directly from code, etc.) This structure is divided into 3 phases for now: preparation, data migration and finalizing (which is something I've never seen before.) Running the prepare phase adds new columns and tables, so the application servers can still access the database. The data migration stage defines what needs to be done, both forward and backwards, which you can also execute in multiple ways (e.g. set up a process that migrates both old and new requests to the new data format.) Then after upgrading your application servers to new code, you can execute the finalize stage to drop old columns, etc. (All of this should happen with monitoring tools and good operation procedures in place, so I'd recommend doing nothing totally automatically.)
Feel free to contact me or open an issue here [1] if you can think of a better way to structure migrations.
Of course you need to do a lot of work yourself still, but at least Storex doesn't fight you, but instead gives you the flexibility to implement your operations exactly the way you want.
Also for the client-server interactions, additional packages will be written to help with this. You're more than welcome to contribute your experience in issues or by contacting me directly to design this in way that accounts for the real-world challenges involved.
What you're saying in the end is also what I'm saying: creating a real-world application is more than just one mega abstraction, but a collection of problems that we need tools for that we can flexibly recombine to our highly individual needs.
[1] https://github.com/WorldBrain/storex-schema-migrations
I understand were you're coming from, I used to think why not. Well the security is still the barrier and even back in the old ages of Java EE the beans stayed safe guarded in the backend from the Web Project.