Ask HN: What was your experience using a graph database?
I have an idea that I want to work on during the break. And I think this is something that would suit a graph db, (a service that would link users together depending on their choice profile). So what was your experience of working with one? Which graph db did you use?
Edit: What sort of problems are well-suited to graph databases? In other words, what are some scenarios where something like Postgres is not suitable anymore?
97 comments
[ 3.4 ms ] story [ 188 ms ] threadBut, it hasn't been around for long and doesn't have any options for hosting, and the JS client library is pretty basic - so you need to do a lot to have something a little more abstracted like Mongoose.
I enjoyed learning something new and will use it again - but unless I need queries that traverse many relationships, I'll probably use Postgres.
We're on Rails and so use the Neo4j.rb gem which has been around for quite a while and also has a ton of work and support around it. The Ruby DSL for it makes it as easy as you would expect in Rails for most basic relationships and queries, and you can access more advanced features or drop into GraphQL as needed.
For our use case, a graph DB was definitely easier than trying to manage relationships and categories in a relational DB but it will definitely depend on your use case. Good luck!
As you say, the appropriate use of a graph database is highly situational. I certainly would not advise doing so without a great deal of careful consideration. However, I tend to doubt whether any amount of consideration would be sufficient; it may be that the most reliable way to determine whether a graph database is the best solution is to try it and see. At least, I was never really able to stop wondering whether I might be able to do things just as easily using Postgres; I would defer to those with more experience, however.
edit: I've previously looked into ArangoDB.com, Dgraph.io, JanusGraph.org, and Cayley.io (to run on top of CockroachDB). I understand all of these are scalable distributed systems, and Postgres is not (CitusData.com aside). Do the benefits of these other systems mainly come when you outgrow single-node Postgres (which has JSONB for "document" storage, PostGIS.net, Timescale.com, etc)?
edit 2: where can I find more technical, concrete examples than https://neo4j.com/use-cases?
If you have a graph, then a graph database, with built-in graph algorithms, will be able to run operations on your graph without pulling all the data out to a client. I'm not an expert in PostgreSQL but I don't think it has any graph algorithms?
How does one know if it's time to migrate to a graph database?
Right now I have a large (6.5GB) database that I have to sometimes work with, and the queries are getting more complex as more features are added to the project.
Right now the data is stored in MySQL, but to get anything interesting out of it requires multiple queries and then sifting through the data later. It all seems cumbersome.
The only time I could imagine that you'd need a specialist graph database would be if you had a very large number of nodes and some time / space intensive algorithm to run over them. Even in this case you could just store the data in a relational DB and use a low level interface (e.g. C) to write the specialist algorithm.
In any case putting the graph nodes and links into sql tables is a much easier to do option.
https://pgrouting.org/
It's about the relationships (between the nodes).
E.g., I frequently need to query, "What is the list of ancestors from the object to the top of the tree?"
In a relational system, this needs to be stored in some kind of data structure, which is redundant. But theoretically in a graph database, it'd be a fast O(log n) query if I'm not mistaken.
Oh and the ability to query for shortest path and similar graph computations, and the DB does all the heavy lifting, is super nice.
https://docs.arangodb.com/3.4/AQL/Views/
- It's miserably slow (if you wish to contradict this statement, please provide numbers) - Consumes gobs of memory (export our data to JSON and it's orders of magnitude smaller) - Full text search will consume all your CPU cores for if you given it a short query (seriously, don't touch this feature, it is a basket full of footguns) - Resource leaks (the Cassandra backend used to leak full databases!)
I've been up to 3:00 a.m. dealing with bugs in Datomic. What use cases does it actually work for?
Also, if you're curious about this sort of database design, NASA shared some interesting work on XDF: TheExtensible Data Format Based on XML Concepts[0][1], which was part of their long-range Constellation Project[2] toolset for building, launching, and operating the Ares spacecraft. They detailed it in this NExIOM slideshow[3], which reading again after quite a while brings back some very good memories. Enjoy!
0: https://nssdc.gsfc.nasa.gov/nssdc_news/june01/xdf.html
1: https://github.com/sccn/xdf/wiki/Specifications
2: https://en.wikipedia.org/wiki/Constellation_program
3: https://step.nasa.gov/pde2009/slides/20090506145822/PDE2009-...
Design your data schema first, then design your queries and finally your data lifecycle pipeline. Run some estimates on the order of magnitude for inserts, query rates, query types and storage sizes - then compare those numbers to the real-world perf of the various graphdb solutions. In general, compared to more typical solutions, you have more expensive inserts, query costs and storage sizes in exchange for more expressive queries. There aren't many application where those cost tradeoffs make sense.
Source: Twice now (2012 and 2018) I've reviewed available graphdbs for storage of enterprise security data when doing the initial platform technology selection. Both times the team fell back onto more traditional approaches.
If I were interested in them again now, I would try new and fancy solutions first, to see if there are nosql-level performance improvements in the graph db space.
neo4j is the most mature solution I found (in the Java space). if you want to use something else go for it, but you may be surprised at the low quality.
op: I strongly recommend implementing most/all of your pipeline using graph & non-graph approaches. choose the graph approach iff you can demonstrate with hard evidence that it makes sense.
Not to detract from your general point, but curious whether you looked at Dgraph in your analysis. It's quite fast and was built for speed.
https://dgraph.io/
Yes! When I was younger I worked on a problem once that needed to compute some very basic graph metrics. My seniors tried to do the work in an early graph database and it was a disaster. It turns out literally just reading in the lines from a file and counting things got the job done in a few seconds.
They refused to use the results until they were coming out of the graph database because "just in case we needed other metrics". We never needed the other metrics.
Calculating over or walking over graphs sucks because there is usually a better, less brute way for any particular query.
Unless you have a set of use cases that require the ability to query across near enough random and unindexable subsets of a graph (eg Facebook), you’ll probably be better off with a DB and a spot of flattening.
See previous discussions on GraphBLAS https://hn.algolia.com/?query=GraphBLAS&sort=byPopularity&pr...
[1] http://graphblas.org
[2] IBM GraphBLAS https://github.com/IBM/ibmgraphblas
[3] Jenna Wise, CMU http://www.cs.cmu.edu/~jlwise/
[4] Formal verification proof of the GraphBLAS C API https://github.com/jennalwise/graphblas-verif
Thanks for the link.
But MIT and Sandia Labs are taking the linear algebra model to the next level, and are now working on encoding an entire operating system in the language of linear algebra...
Jeremy Kepner (the head of MIT Lincoln Labs and GraphBLAS lead) and his team just published a paper [2] where they define an entire unix operating system using the same linear algebra model as they used for D4M/GraphBLAS. The linear-algebra OS model scales linearly way beyond the Linux limits, and since the entire OS kernel representation is defined as generic matrix transformations, it can run on any processor, including CPUs, GPUs, or a cluster of TPUs.
[1] A Linear Algebraic Approach to Datalog Evaluation (2017) [pdf] https://arxiv.org/abs/1608.00139
[2] TabulaROSA: Tabular Operating System Architecture for Massively Parallel Heterogeneous Compute Engines (2018) [pdf] https://arxiv.org/pdf/1807.05308.pdf
https://www.youtube.com/watch?v=xnez6tloNSQ
I've never seen any advantage to graphdbs over relational models until I saw this talk. Raising graph analysis to the level of linear algebra is brilliant.
[1] RedisGraph 1.0 launch video https://www.youtube.com/watch?v=S5WWBzi0LcM
[2] RedisGraph 1.0 https://github.com/RedisLabsModules/RedisGraph
[3] RedisGraph benchmarks https://redislabs.com/redis-enterprise/redis-modules/redis-e...
Adjacency matrices is how graph problems were handled by APL programmers in the 1980s, mostly because there was no alternative before nested arrays. At the time, there was not a lot of vectorized hardware, and main memory sizes were too small for many problems, so the adjacency matrix representation was more of a problem than a good solution. It's really the advances in vectorized hardware and main memory sizes that have made this technique practical for large graphs.
[1] Graph Algorithms in the Language of Linear Algebra, by Jeremy Kepner and John Gilbert https://epubs.siam.org/doi/book/10.1137/1.9780898719918
Datomic (an immutable database for doing functional programming in the database) is central to my startup http://www.hyperfiddle.net/ , I don't think Hyperfiddle is possible to build on other databases that exist today. The future lies in immutability, full stop.
Datomic : databases :: git : version control
I have yet to run into a use case where a graph provided more value than a relational model. I'm sure they exist, but I haven't found them yet.
Neo4j is a great database if you learn how to use it and are willing to get your hands dirty every once in a while (write Java). I keep a blog at maxdemarzi.com on the things you can do with it. See the dating site blog series it may be relevant to what you are doing.
We have thousands of videos, slideshares, blog posts trying to teach graphs. If you take the time to learn you will be successful. If you connect with us on Slack and ask you will be successful.
If you treat it like an rdbms you will fail. See https://m.youtube.com/watch?v=oALqiXDAYhc for a primer and see https://m.youtube.com/watch?v=cup2OyTfrBM for the crazy stuff you can do that most DBs can’t.
No exception today: I used Neo4J at a previous startup, but after using the "free" non-scaled version of it, we got into a hard bottleneck due to a lack of scalability. When we looked to scale Neo4J, we almost had a heart attack when seeing the price. Being this a 50 people "developing country" startup we could not afford to pay the very steep prices.
My faith in it was soured at that point. That said, this was probably 5ish years ago now, so I cannot speak to how much it's stabilized since then.
One of the projects was a business workflow application centered around validating business processes by collecting and reporting on process data—think manufacturing quality control. A graph database was used in an attempt to allow application users more control in defining their workflows and give them more expressive semantic reporting abilities. We tried several graph databases. In reality what happened was that the scheme became implicit and performance was truly awful. The choice of a graph db was a strategic decision; we wanted to enable a different user experience. We probably could have done this project in 20% of the time with a standard database and wound up with a better result.
I have also worked on a problem related to storing and retrieving graph data for image processing. The graph db was obscenely slow and inefficient despite the data models being actual graphs.
Both of the projects I worked on involved people who are experts in graph databases. The level of nuance and complexity was astounding. Even simple tasks like trying to visualize the data became monumentally complex.
My takeaway from both of these experiences was that unless you intend to ask questions about the relationships, a graph might not be a very good fit. Even in that case, other databases will likely perform just as well.
(edit: add last sentence)
They burned a shitload of money on those bad decisions, on that and other products they'd previously stuck on Neo4j for no good reason, which were also seeing poor and unpredictable performance and having a rough time with immature supporting tools for the DB. Whole thing's closely related to the "we have big data, it's in the single digit GB range, so big! We need big data tools!" error, I think.
> My takeaway from both of these experiences was that unless you intend to ask questions about the relationships, a graph might not be a very good fit. Even in that case, other databases will likely perform just as well.
Precisely the same conclusion I reached, at least in the case of Neo4j. If the main thing you need to do is answer questions about graphs, it might be an OK DB to use. If the main thing you need to do is extract data from graphs, then you sure as hell don't want it as your primary datastore. Maybe—maybe—some kind of supplement to a SQL DB or whatever depending on your exact needs, but it shouldn't be what you're actually storing most of the data in.
It was useful in trying to find connections and shortest paths. Project wasn’t build out for reasons unrelated to graph dbs.
[0] http://jupiter.phiresearchlab.org/#/main
I built a prototype with Postgres but there were bottle necks in the IO and querying. Did some research and it turns Neo4j might be better suited.
The app is a beer recommendation system using data scraped from Beer Advocate. It makes recommendations based on the location of the user and beers that the user enters in.
Why a graph database might be suitable for this is that there are no null relationships with Neo4j. In my limited understanding, this means that full table scans don't need to occur for making recommendations based on location.
Supposedly cosine simularity and other recommendation algorithms are built in, so looking forward to using that.
Also I prefer projects that have web UI for queries and visualization