AgensGraph looks great. Especially the ability to execute OpenCypher query or embed it in SQL. If it can be used with citus extension and sharding mechanism we can have distributed graph database which supports ACID.
Also, the official site says it's a commercial product. I wonder how many features are supported in community/opensource/free edition ?
All of this. It's impossible to run Neo4j out of the box in HA in a sane way. Not to mention, you can't even run a reliable backup of the datastore without shutting down the database entirely... unless you pay for their archaic pricing model for enterprise.
We're currently using it and it's got some pros/cons, though the cons are a little scary. Jepsen did an evaluation of Dgraph[0] shortly after we started using it and I can confirm seeing a lot of what was noted. Dgraph is _fast_, but there are problems with the underlying data store (their own DB called Badger). Some of those issues have been remedied, but many still exist.
As a graph database, it has some non-typical tradeoffs. You can't easily discern incoming edges and there's no true node deletion. There's a pretty narrow happy-path where the DB works as advertised/expected, but it's just a fairly young DB from an understaffed startup. Probably worth waiting a year or two for the kinks to be ironed out.
Congrats to Neo4j on the raise! I hope it changes the perception of US VCs w.r.t. graph DBs, who are falling behind the dev enthusiasm and readiness for adoption in this field.
As Jepsen report mentioned, it had identified 23 issues, 19 of which were resolved before the report released and another one right after. Dgraph has gone a long way since v1.0 release in terms of production stability. I'd recommend trying out the latest v1.0.9 release or the upcoming v1.0.10.
Dgraph itself is close to being launched in production at a few very big and well-known companies (that we can't mention publicly yet), who moved away from Neo4j to Dgraph. Needless to say, Dgraph's performance and scalability far exceed any other graph DB in the market.
Dgraph is tackling a lot harder problem of doing distributed joins and traversals, while providing distributed ACID transactions, synchronous replication and linearizable reads. The equivalent of Spanner, which can also do efficient joins (something relational DBs suck at, so technically more complex). There's no graph product out there like this or even a single paper which Dgraph is based on, rather we had to do original research to perfect this technology -- which is why it took time to build and stabilize Dgraph.
Badger, the underlying kv DB, itself was never found to have an issue. It is serving several petabytes of data in production use at various companies. We built Jepsen style bank tests for Badger, which run successfully nightly, and there's an open bounty of $1337 for finding any data loss bugs in Badger.
Dgraph is decently staffed (7 engineers) for a seed-stage startup, but we're definitely hiring and planning to grow in SF. No need to wait, this is the right time to run Dgraph in production.
Regarding performance: have you seen Roi Lipman's RedisGraph presentation (https://www.slideshare.net/mobile/RoiLipman/graph-algebra)? GraphBLAS is elegant and very promising. There some benchmarks comparing RedisGraph to Neo4j and others at the end of the slides. I would love to see Dgraph benchmarked against RedisGraph the same way.
I used to use OrientDB but moved away due to stability issues (long ago now so hopefully they have that sorted). I also just noticed they’ve been acquired by SAP!
For me graph layouts are conceptually superior vs relational when explaining to non techie users.
Practically however I now stick to Postgres - it’s ‘good enough’ (for what I’m doing) and has a heap of benefits in and of itself.
I looked st Agensgraph but I couldn’t get enough info on it, plus it is a custom version of Postgres (not a plug-in) and i think they recently switched to AGPL which makes it overall less exciting to investigate.
I know there are places where having a real graph db helps but I’ve not personally hit those scenarios yet.
Oh man did I struggle with Cypher to start. Conceptually I felt like I understood it but 2/3rds of my queries would go out to lunch and never come back.
Lessons learned for me:
- start with very small datasets (mine was 10k nodes/40k edges/5 types and that still sent it to the moon in many cases)
- get to know the query planner right away
Cypher seems to reward a more deliberate and iterative approach to learning it rather than the ol' grip it and rip it. Once I settled down a bit it came together quite nicely.
Graph databases have been used for YEARS for fraud detection. There was a company called Metatomix out of Boston that did this back in 2008.... I think they got sold to Ab Initio.
Funny to me how tech is constantly a cycle of folks having no idea about the previous incarnations of their ideas.
I've worked extensively with RDF triple stores of various types in the past.
Not impressed. Too academic, too expensive, bad performance.
SPARQL is one of the worst query languages I've seen. We can thank the W3C for that. Using it for a few weeks makes it clear that it was born in an ivory tower committee rather than engineering experience.
Is "nabbing a Series E" really a sign of something flourishing? The way I view investment rounds is as a letter grade. Once you've reached a "Series F" maybe you've just failed? Has this company made any profit, for instance? Just conjecture.
Neo4J is used in production at my company. We have had problems with the clustering architecture known as HA. It was superseded by causal clustering in latest Neo4J releases. We are still requalifying our app with this new system, but we are reasonably confident that it will improve things significantly.
The biggest selling point is that business people have no problem discussing the graph data model with us, data engineers. And their OGM layer is easy to use (but it lacks support for multiple inheritance, at least in its Java version).
I appreciate your response, and forgive me for asking the same question, but do you have a more tangible example? I'm particularly interested in understanding more by seeing something like the disaster SQL for traversing hierarchies and comparing it to an RDF implementation.
Do you have any reference material on using neo4j for complex pricing - we're looking at using neo4j for this, and some evidence that it's going well in other places will increase the comfort level.
Off the top of my head, I know of two big and complex pricing engine projects in production. One is the Marriott one I referenced in a previous post. Scott Grimes spoke about it at GraphConnect 2016:
https://www.youtube.com/watch?v=wdFYEuWWpzo
The other one is confidential but is in the airline industry.
I too would like to hear about getting facts from an sql db, and writing back results from prolog - it's hard to see how a prolog text file can/could be used as a source for transactional problems like booking (making sure the answer you find can be committed while still valid).
I suppose one could have a prolog service that came up with suggestions, ordered by preference, and then attempt committing them in order to the actual transactional storage layer. But I don't see how you could avoid re-implementing transactions.
With an rdms you could look for options, and attempt a booking in a transaction, and have a fairly established way to resolve conflicts.
Yes, but your clustering is just read-only replicas. So you still have the problems that when your graph gets huge and you run out of space to scale each node up you're going to have a bad time. And it also ignores that when you're talking about scaling up to truly monstrous machines with TB of memory the cost is astronomical.
I agree, it's not an easy problem to solve, but saying nobody has solved it isn't accurate either - OpenLink (who I mentioned previously) solved it for their RDF engine. At a previous gig using that across an 8 node cluster we were seeing load rates of hundreds of thousands of edges (or in this case, quads) per second.
22 comments
[ 2.0 ms ] story [ 58.3 ms ] threadAlso MS SQL Server supports graphs natively: https://docs.microsoft.com/en-us/sql/relational-databases/gr...
Also, the official site says it's a commercial product. I wonder how many features are supported in community/opensource/free edition ?
As a graph database, it has some non-typical tradeoffs. You can't easily discern incoming edges and there's no true node deletion. There's a pretty narrow happy-path where the DB works as advertised/expected, but it's just a fairly young DB from an understaffed startup. Probably worth waiting a year or two for the kinks to be ironed out.
[0] https://jepsen.io/analyses/dgraph-1-0-2
Congrats to Neo4j on the raise! I hope it changes the perception of US VCs w.r.t. graph DBs, who are falling behind the dev enthusiasm and readiness for adoption in this field.
As Jepsen report mentioned, it had identified 23 issues, 19 of which were resolved before the report released and another one right after. Dgraph has gone a long way since v1.0 release in terms of production stability. I'd recommend trying out the latest v1.0.9 release or the upcoming v1.0.10.
Dgraph itself is close to being launched in production at a few very big and well-known companies (that we can't mention publicly yet), who moved away from Neo4j to Dgraph. Needless to say, Dgraph's performance and scalability far exceed any other graph DB in the market.
Dgraph is tackling a lot harder problem of doing distributed joins and traversals, while providing distributed ACID transactions, synchronous replication and linearizable reads. The equivalent of Spanner, which can also do efficient joins (something relational DBs suck at, so technically more complex). There's no graph product out there like this or even a single paper which Dgraph is based on, rather we had to do original research to perfect this technology -- which is why it took time to build and stabilize Dgraph.
Badger, the underlying kv DB, itself was never found to have an issue. It is serving several petabytes of data in production use at various companies. We built Jepsen style bank tests for Badger, which run successfully nightly, and there's an open bounty of $1337 for finding any data loss bugs in Badger.
Dgraph is decently staffed (7 engineers) for a seed-stage startup, but we're definitely hiring and planning to grow in SF. No need to wait, this is the right time to run Dgraph in production.
I used to use OrientDB but moved away due to stability issues (long ago now so hopefully they have that sorted). I also just noticed they’ve been acquired by SAP!
For me graph layouts are conceptually superior vs relational when explaining to non techie users.
Practically however I now stick to Postgres - it’s ‘good enough’ (for what I’m doing) and has a heap of benefits in and of itself.
I looked st Agensgraph but I couldn’t get enough info on it, plus it is a custom version of Postgres (not a plug-in) and i think they recently switched to AGPL which makes it overall less exciting to investigate.
I know there are places where having a real graph db helps but I’ve not personally hit those scenarios yet.
Lessons learned for me:
- start with very small datasets (mine was 10k nodes/40k edges/5 types and that still sent it to the moon in many cases)
- get to know the query planner right away
Cypher seems to reward a more deliberate and iterative approach to learning it rather than the ol' grip it and rip it. Once I settled down a bit it came together quite nicely.
Funny to me how tech is constantly a cycle of folks having no idea about the previous incarnations of their ideas.
https://tinkerpop.apache.org/gremlin.html
I suppose one could have a prolog service that came up with suggestions, ordered by preference, and then attempt committing them in order to the actual transactional storage layer. But I don't see how you could avoid re-implementing transactions.
With an rdms you could look for options, and attempt a booking in a transaction, and have a fairly established way to resolve conflicts.
I agree, it's not an easy problem to solve, but saying nobody has solved it isn't accurate either - OpenLink (who I mentioned previously) solved it for their RDF engine. At a previous gig using that across an 8 node cluster we were seeing load rates of hundreds of thousands of edges (or in this case, quads) per second.