Awesome news! Did not know know about this new investment as a remote-working member of the team until now. Looking forward to the future, what we can achieve in the coming months.
Why will ArangoDB succeed where RethinkDB failed commercially? Can't see any mention of changefeeds on the Arango website, which is the killer Rethink feature for me.
ArangoDB is a multi model store, which is to say it lets you store key value pairs, json documents, and graphs. (Edges are just another document with two special fields: "_from" and "_to".
My team just spent 3 weeks investigating document stores and ultimately settled on arangoDB because: 1. Graph functionality, 2. Read after write consistency, 3. Partial updates, 4. Efficient joins across shards, 5. The (limited) text search features, 6. AQL is just pretty sweet. There was a whole spreadsheet matrix but that's what I can remember without pulling it up.
We really liked RethinkDB for its change feeds. We are probably going to use Elastic for search, because we want to expose Lucene to our users, so we'll probably have to write some foxx endpoints to manage the data replication. I like the idea of foxx but really hate that it's JavaScript based. I'm planning on looking into using ScalaJS if possible.
It should be possible to use ScalaJS with Foxx as long as it is transpiled in advance and produces node-compatible modules (i.e. something that can be consumed by and can consume other node modules using `require`). I've heard of other people successfully using Babel, TypeScript and even ClojureScript with Foxx.
Aw yeah! While I have you - has any progress been made on a correct write ahead log APIacross shards? I saw a post from mid 2015 about it, but didn't see anything after. Would really like to be able to write an elastic river-type tool for search.
I'm not entirely sure what you mean. If you can point me in the direction of the article you mentioned, maybe one of the developers who are more intimately familiar with the cluster internals can take a stab at it?
The question makes more sense in context, but I'm not familiar enough with the WAL to be able to answer myself. When you're back on dry land, maybe you can join us in the Community Slack: https://slack.arangodb.com
We have a dedicated #cluster channel were you will probably have better luck getting detailed answers to technical questions. :)
I have not used Arango in production although I evaluated it and exchanged a few emails with Claudius.
I see a number of reasons why ArangoDB might be more interesting than Rethink:
1) Arango is a multi model database with graph, noSQL, and key-value pairs. Even if Rethink has joins and complex queries, it will never be as flexible for general data modelling.
2) Foxx is a logical layer on top of the database. While it's limited compared to a classic server side application it offers way more flexibility than Rethink could ever.
3) Arango (or Foxx) don't offer realtime data like Rethink or Firebase, but quite frankly there are few cases where such feature is needed.
On our project we are probably going to use DreamFactory instead of Arango because we are more comfortable with SQL dbs and it has a number of other valuable features over Arango.
CouchDB's changes feed predates Rethink, I'm not sure why it largely gets ignored. Couch is rock solid.
For graph DBs, I've found nothing easier to setup and more efficient than Arango. The team there is very receptive and determined too.
I'd love it if Arango had a changes feed, but realistically it's really easy to just pipe the data into a stream from whereever gets confirmed writes to the database. This is how I've always done it outside of Couch/Rethink. It helps decouple the stream from the db concept in case more data sources or sinks are added later (very common in my experience).
Realm has the best implementation of live realtime updating objects/queries that I have tried, but the model is bit different from databases like Rethink in that it is an embedded database that you run locally (but which can then live sync with other instances).
I'm an ArangoDB team member. We are starting with our own Jepsen tests. But we will ask Aphyr as soon as possible to do an official test. In the meantime can the community check our implementation.
It most certainly makes sense to jepsen Arango before asking the public to do so. But I won't take your tests without a grain of salt till Aphyr does so. ;) Best of luck
Jepsen tests are basically a closure framework to conduct such tests. We took some of the existing test cases and rewrote them for ArangoDB. So these are not "official Jepsen tests" but tests we did ourselves to track down issues. If you want to inspect and retest yourself, that will be possible. There's no reason to keep the tests closed source. Of course it would be great to get ArangoDB Aphyr-approved eventually.
What is it that worried you? That the team is going to check things over to make sure that they have things done right?
The only alternative to that is to just kind of pretend that it all works without doing the hard work to verify. You saw how that ended up for a lot of other systems that Kyle has tested.
We are closely monitoring the development. Not being a Java database, we cannot run Java queries natively. Tinkerpop3 has added some hooks for non-Java, but it would still very hard to get the speed of AQL. But I hope that Tinkerpop will open up to non-Java even more.
We're watching node's support for the webkit developer tools closely but as their recent releases show, integrating with external debugging tools is difficult because V8 is always a moving target. There are also architectural differences that make it more difficult to offer debugging capabilities in Foxx, such as node being typically run as a single process whereas a typical ArangoDB installation will use any number of parallel V8 threads and incoming requests being load balanced accordingly.
We're constantly trying to improve the developer experience of Foxx, though, and the partial rewrite of Foxx for ArangoDB 3.0 was a big part of that, hopefully reducing the need for debugging by making the API easier to reason about.
During development I would recommend trying out the development mode and syncing your changes to the deployed service on the file system. For debugging queries it can also be helpful to run them from the web interface first and then use `aql` template strings in JavaScript instead of constructing the queries piece by piece.
It takes time and money for an open-source DB to become established. I think Arango is taking the right steps and this investment will help a lot in getting there.
PS: I only started using ArangoDB recently for the first time on a client project and it was refreshing to say the least. Being able to do complex joins, coming from a Mongo environment feels so much better.
why do we need this? Mongo (3.4) is now doing graph and will also (obviously) do document and key-value. Is there something special about the storage engine, for example, that will make this awesome for range queries (contiguous location ala RIAK TS or Cassandra rows)? Does it optimize for the new SSD controllers that have order of magnitude higher random read throughputs? Does it leverage GPU RAM? Does it occupy some barren area of the CAP triangle?
Does it do something new, or is it just another neat query language on top of the same old storage algos? The "3-in-one" pitch doesn't fly for me unless the underlying storage tech does something special.
I'm an ArangoDB team member. The new graphLookup is far away to be a graph functionality. It's only a recursive lookup.
Yes, the storage is special. The three data models are combined in one database core.
I'm curious on the graph database claims - how is this different to MongoDB using _from/_to attributes to represent edges? It certainly looks better than MongoDB in all other aspects (though to improve on MongoDB would not be difficult!).
How will it get the performance of a Neo4J,Titan et al. with this model? Does it index them specially?
Seems odd to ignore Gremlin and OpenCypher just to have a proprietary / "yet another standard" graph query language.
One major difference is that ArangoDB has full support of transactions across collections. Also while MongoDB now has limited graph functionality, ArangoDB is actually optimized for graph queries and can do arbitrary traversals and joins very efficiently.
The problem with Gremlin/OpenCypher is that they are primarily designed as graph query languages. ArangoDB takes a multi-model approach and many ArangoDB users focus on documents first and add in graph queries as an optimization step as their products mature. AQL is designed to be very easy to learn and mixes the document and graph paradigms seamlessly, embracing the multi-model nature of the database.
Personally I think that starting with a blank slate was the right choice compared to other multi-model databases that have found it necessary to extend a subset of a standard query language with their own proprietary additions to achieve the same goals.
We did a benchmark that shows you the performance regarding Neo4J. Before you shout that it made by us have a look, please. It's all open source. The data, the scripts and so on. Everything we use for it, we described in detail.
AQL is much more than a graph query language; therefore, Gremlin and OpenCypher aren't alternatives.
46 comments
[ 3.4 ms ] story [ 111 ms ] threadMy team just spent 3 weeks investigating document stores and ultimately settled on arangoDB because: 1. Graph functionality, 2. Read after write consistency, 3. Partial updates, 4. Efficient joins across shards, 5. The (limited) text search features, 6. AQL is just pretty sweet. There was a whole spreadsheet matrix but that's what I can remember without pulling it up.
We really liked RethinkDB for its change feeds. We are probably going to use Elastic for search, because we want to expose Lucene to our users, so we'll probably have to write some foxx endpoints to manage the data replication. I like the idea of foxx but really hate that it's JavaScript based. I'm planning on looking into using ScalaJS if possible.
It should be possible to use ScalaJS with Foxx as long as it is transpiled in advance and produces node-compatible modules (i.e. something that can be consumed by and can consume other node modules using `require`). I've heard of other people successfully using Babel, TypeScript and even ClojureScript with Foxx.
If it makes sense to take off thread, my email is daniel.porter@pwc.com. Thanks so much for sharing your knowledge!
We have a dedicated #cluster channel were you will probably have better luck getting detailed answers to technical questions. :)
I see a number of reasons why ArangoDB might be more interesting than Rethink:
1) Arango is a multi model database with graph, noSQL, and key-value pairs. Even if Rethink has joins and complex queries, it will never be as flexible for general data modelling.
2) Foxx is a logical layer on top of the database. While it's limited compared to a classic server side application it offers way more flexibility than Rethink could ever.
3) Arango (or Foxx) don't offer realtime data like Rethink or Firebase, but quite frankly there are few cases where such feature is needed.
On our project we are probably going to use DreamFactory instead of Arango because we are more comfortable with SQL dbs and it has a number of other valuable features over Arango.
Meteor also has something called "reactive data sources" although Meteor is not really a DB.
As far as realtime and multi-model (graph) I work on https://github.com/amark/gun , which is also distributed and offline-first.
For graph DBs, I've found nothing easier to setup and more efficient than Arango. The team there is very receptive and determined too.
I'd love it if Arango had a changes feed, but realistically it's really easy to just pipe the data into a stream from whereever gets confirmed writes to the database. This is how I've always done it outside of Couch/Rethink. It helps decouple the stream from the db concept in case more data sources or sinks are added later (very common in my experience).
[0]: https://www.arangodb.com/2016/05/getting-closer-arangodb-3-0...
[1]: https://github.com/arangodb/jepsen/commit/2b79809df73ca6c755...
The only alternative to that is to just kind of pretend that it all works without doing the hard work to verify. You saw how that ended up for a lot of other systems that Kyle has tested.
Is there plan to support gremlin query natively ?
We're currently using Arango and it gets tedious to call foxx-manager replace every time I'm doing a change on the db
We're constantly trying to improve the developer experience of Foxx, though, and the partial rewrite of Foxx for ArangoDB 3.0 was a big part of that, hopefully reducing the need for debugging by making the API easier to reason about.
During development I would recommend trying out the development mode and syncing your changes to the deployed service on the file system. For debugging queries it can also be helpful to run them from the web interface first and then use `aql` template strings in JavaScript instead of constructing the queries piece by piece.
PS: I only started using ArangoDB recently for the first time on a client project and it was refreshing to say the least. Being able to do complex joins, coming from a Mongo environment feels so much better.
Does it do something new, or is it just another neat query language on top of the same old storage algos? The "3-in-one" pitch doesn't fly for me unless the underlying storage tech does something special.
How will it get the performance of a Neo4J,Titan et al. with this model? Does it index them specially?
Seems odd to ignore Gremlin and OpenCypher just to have a proprietary / "yet another standard" graph query language.
One major difference is that ArangoDB has full support of transactions across collections. Also while MongoDB now has limited graph functionality, ArangoDB is actually optimized for graph queries and can do arbitrary traversals and joins very efficiently.
The problem with Gremlin/OpenCypher is that they are primarily designed as graph query languages. ArangoDB takes a multi-model approach and many ArangoDB users focus on documents first and add in graph queries as an optimization step as their products mature. AQL is designed to be very easy to learn and mixes the document and graph paradigms seamlessly, embracing the multi-model nature of the database.
Personally I think that starting with a blank slate was the right choice compared to other multi-model databases that have found it necessary to extend a subset of a standard query language with their own proprietary additions to achieve the same goals.
We did a benchmark that shows you the performance regarding Neo4J. Before you shout that it made by us have a look, please. It's all open source. The data, the scripts and so on. Everything we use for it, we described in detail.
AQL is much more than a graph query language; therefore, Gremlin and OpenCypher aren't alternatives.