132 comments

[ 2.9 ms ] story [ 192 ms ] thread
At last, an open source alternative to Firebase.
Great move, RethinkDB folks!

I'm looking forward start prototyping some apps using Horizon. On a first look seems very promising.

Congrats!

I'm continuously impressed by Rethink. When I first used it, I ran into a few issues with changefeeds. Since they work in the open, I could quickly look at the Github issues and see that the fixes/features were already slated for future release milestones. To me, that's a huge win.
I love Firebase, but this is very exciting. Congrats on 1.0!
Hey guys, Slava @ Rethink here. The team is really excited to launch Horizon -- it's based on a lot of feedback from users of very different backgrounds, and we think it will make web development dramatically easier.

I've been up for about twenty-four hours, but I'll be around to answer any questions for the rest of the day.

Hi Slava!

What do you guys think about the approach taken here, and what are the major differences in approach theat you took with Horizon?

https://github.com/mikemintz/react-rethinkdb

I've had really good luck with that whitelisting approach, so I'm wondering what the biggest wins would be from potentially switching to Horizon.

Mike's `react-rethinkdb` project is really cool, but it's based on exposing the entire ReQL language to the browser. We thought this approach wouldn't work well as an industrial-strength product -- ReQL is designed to be accessed from the backend, and exposing a full query language to the browser introduces a number of challenges. Security is really hard, learning curve is steep, there are DoS challenges, the protocol is too complex to write platform-specific extensions, etc.

On the other hand, Horizon exposes a simple API/protocol that's easy to learn and easy to secure. If you need more functionality, you can import Horizon as a Node.js module and start writing backend code that accesses RethinkDB directly. The learning curve is much easier, it provides an upgrade path for sophisticated apps, and feels like a more robust product.

(I don't mean to disparage `react-rethinkdb`, I think the project is awesome, it's just a philosophical difference in approach)

Thanks for your time!

Could you give me a quick comparison of Horizon and Meteor? I've used Meteor in the past (and loved it), and it looks like Horizon might be fun to learn.

There's one on the website:

http://horizon.io/faq/

IMO it hand waves Meteor away without acknowledging (perhaps out of ignorance) some of it's other strengths.

I'll watch Horizon but I'm taking it all with a grain of salt. Standard new javascript framework protocol people.

The FAQ repeats some things I've heard about Meteor -- specifically that its architecture doesn't scale.

But Meteor isn't completely prescriptive. There are several pieces that can be swapped out: You can use Angular, React, or Blaze on the client, for instance.

From what I've read, it doesn't look like Horizon is a complete framework, just a database client/server component that can be used with any framework.

We have a short-burst, write-heavy, mostly cpu-bound, work-load, which according to the various financially incentivized critics would completely knee-cap us for our use of Meteor.

Except we:

1) optimized our database operations

2) relegated cpu-bound work to micro-services that could process work from a queue at their own pace

3) Leveraged the ecosystem to writer smarter and more concise code

Know your enemy and know yourself and you need not fear the result of a thousand battles (Sun Tzu). Meteor is an amazingly productive ecosystem for writing real-time applications. No it will not solve every problem you throw at it.

Just to be clear I'm not hating on Horizon. I definitely will try it in the future.

OK, glad to hear that it does work for you. Sounds like you have a robust architecture.

I don't actually know what is supposed to "not scale" about Meteor. I've just read it multiple times. But because of the nature of Meteor (Really easy for anyone to set up! Anyone can create a real time app in ten minutes or less!), it probably attracts a lot of junior developers. It therefore should have occurred to me to translate "It doesn't scale!" to "I don't know how to scale it and my app is slow!!!". :)

There are two major issues that crop up in my experience.

* Node as a runtime is deeply mediocre, in my experience. You can usually solve this by throwing more (usually virtualized) hardware at the problem.

* Meteor's architecture is (at present) tied into MongoDB's oplog tailing. You can scale it up through sharding, through alternate libraries, and DDP itself is not tied to MongoDB at all. IIRC fixing this was a goal for Meteor 2.0.

Both of these can be worked around. If nothing else, the Meteor web side will talk to anything doing DDP, and DDP is not that hard to deal with.

It comes up because Meteor gets slow faster then most other solutions will get slow. It kind of reminds me in Django how a common mis-step for beginners is using an O(n^2) algorithm on the Django ORM. Meteor is the same way if you aren't using proper indices, or are trying do something that SEEMS doable to the uninitiated (like say pushing ~n thousand JSON objects into a MongoDB set), it's going to give you a bad time.
"2) relegated cpu-bound work to micro-services that could process work from a queue at their own pace" doesn't this defeat the primary selling point of meteor?
The three key features we derive from Meteor are:

1) real-time view layer

2) diverse and highly integrated ecosystem (libs are written just for meteor)

3) productive development environment (es2015 modules on client/server, no fuss build pipline for sass, jade, angular-templates, angular dependency injection)

There are a ton of secondary benefits, but thats the crux of it. Yes we do sacrifice some of the benefits of Meteor as a monolithic (homogenous?) full-stack solution, but I would argue that they are in fact small, and if you dive into Meteor you'll find it to be pretty modular.

It looks to me like Horizon and Meteor are solving problems in different scopes of the application stack.

Horizon looks to be more like the Minimongo/DDP/Accounts parts of Meteor, and doesn't help with any more of the stack like Meteor does (eg: In Horizon its more up to you to wire it into Angular/React/etc). Meteor is more of a platform and Horizon is more of a real-time client DB and account framework.

If I had to choose between the two, Horizon would be good for if you had a strong opinion/technical need for very specific coupling between UI and DB, and Meteor for when you don't or need to get something out very quickly (eg: this weeks internal CRUD app for a few dozen users).

Congrats on the release! I've only hada little time to play with it so far as a beta user, but was super impressed and can't wait to use it on a real project soon. Cheers!
Are the transactions optimistic? How does the logic that concurrently applies/rolls-back these transactions over multiple clients work? I'm very interested in this, do you have some internal documentation on it?
Take a look at this comment: https://news.ycombinator.com/item?id=11716145. TL;DR: currently not optimistic but will be soon; it's fairly simple to do because RethinkDB core has built in support for it. Once the feature lands we'll talk about the implementation in detail.
(comment deleted)
Okay I can wait :)

But without optimistic transactions, what sets Horizon apart from just running the transaction-code on the server? If I understand it correctly, the latency will be the same in both cases, i.e., one network round-trip.

Ease of use.

Lack of optimistic updates is a temporary state, though. The vision for Horizon definitely includes this functionality, and we'll make sure it lands ASAP.

> I've been up for about twenty-four hours, but I'll be around to answer any questions for the rest of the day

Make sure you don't regret about your answers later :) Answering questions on HN after being 24 hours awake is generally not a good idea :)

Hey Slava. Interesting stuff - two questions:

1. Do you have something analogous to https://www.firebase.com/docs/web/api/ondisconnect? Real-time apps often have some manner of presence indicator. Firebase allows you to say "when this client disconnects, set the value of this key to $BLAH". I noticed that Horizon has http://horizon.io/api/horizon/#ondisconnected, but this is just a client-side indication of when you've disconnected.

2. What are Horizon's transaction properties? How about ordering of appends to a collection?

There is currently no "disconnect-and-set" functionality, but it's on the roadmap and will happen soon. It just didn't land into v1.

Since Horizon is backed by RethinkDB, it's transaction properties are the same as that of RethinkDB. Check out this document for the specifics -- http://rethinkdb.com/docs/consistency/.

Once you guys made it big, you should rename yourselves to RethoughtDB :)
Does it handle conflict resolution, ike when two people modified the same field at almost the same time?
Edit: While Horizon assigns versions internally, it looks like they are not currently used for catching concurrent client-side modifications by different users. (I previously wrote that they would cause the "losing" write to fail if it was based on an outdated version of the data, but that doesn't seem to be the case yet.)

I opened an issue against our docs to add information on this, as I'm a bit fuzzy on the detailed behavior myself: https://github.com/rethinkdb/horizon-docs/issues/50

Hi, Slava, I was a web developer and now I am a mobile developer. Nobody is developing web applications anymore (at least in developing countries). We are all mobile first. How does android or ios developers take advantage of Horizon ? Can you give a few examples? How does horizon make mobile (native) developers life easier ? web development is slowing down and mobile development is on the rise. Mobile developers don't want to deal with backend development. But it seems like web development is your main focus and mobile is a second thought. Please make it the focus of your work. Meteor is useless to us because it is optimized for web client. I hope Horizon won't fall for this trap. Your bigger audience and potential users are android or IOS developers. Please optimize for them and provide lots of examples to show them how their life can be easier. Thanks.
I'm from a developing country too and that is such an exaggeration. Maybe your career has been focused on mobile, but no way there's overall more native development than web.
So what does your dumb mobile client connect to? If it's any good, probably a web backend.
Web developer here. Some of the most complicated application are being built for web vs mobile. Within the enterprise web applications, I fail to see how mobile is anywhere near web's presence especially for data intensive apps.
In the FAQ page http://horizon.io/faq/

Can I use Horizon on mobile devices?

Horizon ships with a JavaScript client library that can be used for building web and mobile apps with frameworks like React, Angular, or React Native. We’re also working on specific platform client libraries that support the Horizon protocol. Client libraries for iOS and Android are coming soon.

Very fitting handle.

Im really stoked for this and thanks a lot for your hard work.

Congrats! I remember seeing someone from your team doing a demo at an SF JavaScript meetup. Looks very interesting.
Ive been looking forward to this for so long!!
From their blog post:

> Horizon isn’t prescriptive or particularly opinionated–it’s designed to work well with the JavaScript frameworks that you already know and love.

It is seriously a breath of fresh air to read something like that. Bravo, Horizon/RethinkDB team.

> Horizon is distributed as a standalone server for browser-based JavaScript apps. When you're ready to write custom backend code, you can load the Horizon modules directly in Node.js

This is key. Firebase seems great for quick prototypes, but I never built against it because it was unclear (to me at least) how to deal with cases where you want to leave the sandbox and add custom backend functionality that they haven't built out for you. Horizon seems ideal -- it gives you the ease of use of a turnkey system starting out, with the flexibility to customize backend logic/validation later when you need to. That, plus the fact that it's put together by the awesome RethinkDB team, makes it very interesting to me!

Firebase does support a REST API that I've used successfully from Rails. So back-end access works pretty well.
I've setup background workers that connect to a firebase DB to perform business logic / backendy type stuff. It works rather well and they even have a queue collection specifically for this type of thing. You could even setup a daemon on a VPS / EC2 that would in turn fire up AWS lambda procs (node or w/e) to perform w/ your firebase data.

Regardless, I'm very excited to play around w/ Horizon as well.

Saw a demo of this at a realtime meetup in Palo Alto and was pretty impressed. Like everything RethinkDB does I anticipate it will be a fantastic one-two punch of sound engineering and smooth user experience.
Awesome! Was it Mike Seid's demo? Will have video up soon. Thanks for stopping by our meetup :D
Could you post it here when it's up? I'd like to see it.
Although this is about horizon, if you haven't looked at RethinkdB you should definitely check it out.

It's an awesome database for a variety of use-cases.

Personally, I think their query language + support for joins makes it much more useable than MongoDB (which I've used quite a bit).

Agreed! I had tons of fun building my own CMS on top of RethinkDB (as well as a logging/journal app for personal use). The only reason I ended up with a wordpress/MySQL solution for the CMS was that building the CMS interface is a bitch and migrating from SQL to SQL was a bit easier.

But man was I sad to leave RethinkDB for that!

I think it's amazing that it's working with real RxJS streams for changes now - given how Redux store now returns an observable thing, Cycle is Rx based and Angular 2.0 uses a lot of Rx.

This guarantees that things will interop nicely with eachother and we'll be able to use Rx all around.

You might want to check out lovli.js (https://github.com/flipace/lovli.js) -- a Horizon+Redux integration. It's really cool and demonstrates how well Horizon can interop with modern web technologies.
Oh, this is exactly what I want to do with Horizon! So cool.
How does this differ from Meteor?
See the FAQ http://horizon.io/faq/

For convenience, this is the section taken from it that addresses this exact question! (I imagine other people may be thinking this)

How is Horizon different from Meteor?

Horizon has philosophical and technical differences with Meteor that will result in vastly different developer experiences.

Horizon is a small layer on top of RethinkDB with a narrow, purpose-built API designed to make it very easy to get started building realtime apps without backend code. Horizon isn’t prescriptive – you can use any front-end framework without any magic/customizations, and once your app outgrows the Horizon API you can use any backend technology (e.g. Node.js, Python, Ruby) and any backend framework (e.g. Rails, Express, Koa, etc.)

By contrast, Meteor is a much more prescriptive framework. Horizon is a reasonably small component that has a very clean separation with the database and the frontend, while Meteor is a much more monolithic experience.

Another major difference is architectural – Meteor uses a LiveQuery component built by tailing MongoDB’s oplog. This approach is fundamentally limited – it’s impossible to do many operations efficiently, and even the basic functionality is extremely difficult to scale.

Horizon is built on RethinkDB, so the LiveQuery functionality is in the database. This allows for much more sophisticated streaming operations, and scalability is dramatically simpler because the database has all the necessary information to allow for a scalable feeds implementation.

Slava, Horizon looks cool, and I'm a huge fanboy of RethinkDB. However, in the demo video you write queries client side. How do you protect against users modifying front-end JavaScript and thus the queries?

    // ex
    this.props.horizon.order('datetime', 'descending').limit(8).watch()
This is the question I always get stuck on with these "front-end only" frameworks. How do you prevent DoS attacks or scrapers downloading your whole database? What is the information security model?
Out of the box Meteor does publish the to the browser. But I believe if you remove the insecure package you then control what is published to the browser e.g. usually only a subset of the db.
On the Meteor App I'm currently working on, we limit the data the client sees to what they need, so while it makes a scrapers job slightly easier as they can query the DB, they don't get anymore data than any other approach. If data is displayed to the user they can scrape it.

Security is the same as the clients DB writes are rejected if they are modifying fields in ways that are not permitted (eg: changing a trouble ticket from "submitted" to "approved"). This can make some of the security easier to audit as most of the business logic security is in one place rather than scattered about in dozens of API calls.

While the exact technical details for these real-time DB based front ends are different, the basic thinking is the same of never trust the client and its costly to transmit info to the client.

The nice thing about having the DB directly on the client is that it makes CRUD apps much easier to make as you have to build less bespoke framework to get data to/from the client and verifying things conform to business logic. If you are not building a CRUD app you will want to evaluate other methods for a better fit.

I would think you'd do this server-side, I've had similar issues outside of JavaScript: users play a game and create a custom client that connects to my server, they send custom packets thereby I have to make sure that they: don't access too much data at once (a user asking for 100 different things in 3 seconds is not even normal in certain scenarios) or the data they're requesting is formatted wrong (a string where a integer is expected?) the solution I guess is server-sided. This issue isn't unique only to client-side javascript, it is relevant in networking as well. So long as the back-end is designed intelligently it could all work out.
(Josh from RethinkDB here)

The sibling comment by ubercore has the right link. And your concern is important, since without security, why not just expose your db to the frontend directly?

At a high level, the permission system:

1. Is a whitelist approach (you can't do anything by default)

2. Allows you to specify query shapes that are allowed to be run by the current user. This prevents queries of the wrong shape from ever hitting the database, helping with scalability.

3. Has a fallback to full javascript rules which can depend on the data on disk. A little slower than the query rules, but allows much more flexibility where you need it. And you can combine them with a query rule to do a rough first pass of what is acceptable.

Ultimately, the permissions system is designed to prevent unfettered database access, but it also has an eye towards performance in the case of DOS attacks. Horizon overall attempts to use only indexed queries, low resource usage operations etc, so you don't accidentally invite DOS attacks .

Now, I'm not claiming we're DOS proof or anything, but I just intend to mention we have an eye towards it when we talk about security.

You can't protect from the client modifying the JavaScript code, and that's ok. Horizon ships with a full security model (http://horizon.io/docs/permissions/) that lets you specify exactly what data each user can and cannot access, which queries they can run, etc. We're going to continue expanding the security model to include DoS protection, and any other issues that may come up.

So while the user can change JavaScript as they please, that's ok; if they try to do something they're not supposed to do, the server will just reject the query.

Congrats guys. As usual, amazing work!
How does access control work? For example, if I have a database containing a table "users", and a table "files", and the files table contains a column "owner" (pointing to the users table), then how do I prevent the client-side database engine from accessing the files which do not belong to the user that is currently logged in?
Check out the docs on security and permissions -- http://horizon.io/docs/permissions/. Horizon ships with a full security model that allows you to protect your data and specify exactly what each user can and cannot access and which queries they can run.
(Former Meteor core dev here) This is cool!

Does Horizon also solve "optimistic updates"? If so I'd love to learn more details. For comparison, Meteor keeps a local datastore that updates immediately when data is mutated and then reconciled with the real database.

The current version of Horizon doesn't do optimistic updates, but it's on the roadmap. Check out https://github.com/rethinkdb/horizon/issues/23. This is relatively easy because RethinkDB itself has support for notifying the client via a feed when a particular update has landed (i.e. the RethinkDB client can correlate the write with a feed message). It just didn't make it into v1, but will happen soon.
so .. can one write a spreadsheet app that is able to show results reactively in the front-end and will still reflect that same logic in some kind of backend api ?

Kind of being able to have the same logic in my spreadsheet as well as my api ?

This could be huge!

(comment deleted)
Congrats on the launch, this looks like an amazing product that made immediate sense to me. I have a feeling the Ionic community is going to be quite drawn to this over time.

Are there any dockerized versions of this yet?

For apps that don't necessarily have the realtime requirement, Kinto [1] is another really cool open-source backend solution that focuses on the offline-first scenario. It's built by the Firefox Sync team and used in production at Mozilla.

That said, at first glance there doesn't seem to be anything stopping anyone from building a caching layer on top of Horizon to enable offline-first functionality. Definitely going to be keeping an eye on this as well!

EDIT: It looks like they are in fact considering offline-first support already [2]!

[1] http://kinto.readthedocs.io/en/latest/overview.html

[2] https://github.com/rethinkdb/horizon/issues/58

How is Kinto vs Horizon different, anyone from the Horizon team want to comment.
Offline is much harder than one initially think. More specifically, the big issue us that you almost certainly need a solid algorithm to solve conflicts in order for it to work, otherwise it's quite likely you'll update your data in a matter that is unexpected from a user perspective (assuming that two users edit the same data). The only systems i know of which seem to handle this gracefully is Derby/ShareJS (using operational transformation but still lacking offline support) and Amelisa (using CRDTs and in a very early phase development wise). If this is not built in from the ground up, I fear it's going to be very complicated to build in.
Offline is much harder than one initially think. More specifically, the big issue us that you almost certainly need a solid algorithm to solve conflicts in order for it to work, otherwise it's quite likely you'll update your data in a matter that is unexpected from a user perspective (assuming that two users edit the same data). The only systems i know of which seem to handle this gracefully is Derby/ShareJS (using operational transformation but still lacking offline support) and Amelisa (using CRDTs and in a very early phase development wise). If this is not built in from the ground up, I fear it's going to be very complicated to build in.
I'm really excited to see a framework make security and authentication part of it's foundation.

Amazing work Horizon team!