I used Graphviz years ago to generate diagrams for NLP structures, and other visualizations. It was the only thing that could do what I needed. I ended up moving to Processing/P5 to create "spring physics" interactive graphs, then wrote my own eigenvector embedding visualization that could project into 3D space and had a moveable camera so you could fly around with WASD and mouse. Cool stuff! Graph visualization and 3d graphics was a fun rabbit hole
Do you happen to have any open source code for this that I could look at. I am looking to do some interactive graph viz and would love to see if there is something I could reference
Basically any sketch that has "3d" or "aspekt" in the title is on that track. But you can also see from the thumbnail many of them are 3d world related. The navigation of that site is pretty tricky...but you can find the source code by:
- Click on a sketch
- Click on the 'abacus' control menu to the right
- Click on the files tab in that
Source code in Java should be available there.
Please remember it requires a version of Processing in Java to work. I worked on this in 2011/2012 so you may need to figure out which Processing version that is.
Also, here's a youtube video of flying around one of my sketches:
I used this to generate relationship diagrams between companies, adding in projects and who is working on what; the final image was a map of an ecosystem of that set of businesses.
My single biggest issue with Graphic is that the quality of Arrow end-points is very poor. They don't terminate cleanly at the destination. This is visible in the 2nd link you've posted.
I'm amazed that this hasn't been addressed. It's been there for years.
Makes the end products impracticable for distribution in a professional setting. It's Ok if you're the only consumer ... not great for other cases.
I remember this issue being raised, I think on the graphviz Discourse, and the graphviz developers responding to it as if it was a hard to solve limitation of graphviz for some reason. But I can't for the life of me remember the details.
Ostensibly each arrow style graphic would need an explicitly-defined "attach point" at whatever is deemed the "tip", so that the tip can be oriented perfectly in relation to the vertex and edge of the GraphViz graph.
If nothing like that exists, I can see how it would be a very troublesome refactor.
"Very poor" seems harsh, but, yes, it hurts us, too. (That, and text with slightly-off baselines.)
The relevant code starts around https://gitlab.com/graphviz/graphviz/-/blob/main/lib/common/... If this isn't right, maybe somebody can figure out why. Perhaps the loop stops sometimes on the wrong side of the boundary? Anyway, it's equally possible that when the endpoint coord is handed off to a lower level driver, the arrowhead mitering is wrong. There is no question there was once upon time explicit code to try to cope with this problem, at least in the native PSgen, but I can't find it now. Maybe it wore away as waves of open source development washed over it, along with static tables for a bunch of "standard" PS fonts.
I think I found at some point that the arrows could be made correct -- at least for the SVG rendering in Firefox -- by removing the stroke (stroke-width="0").
I never had this problem. Arrow style is configurable if I remember correctly. You can configure (extend) also overlapping in boxes so the arrows never enter into the box area.
I'm talking by heart and I could be wrong in any case.
1) is there an option to make the graph horizontal instead of vertical?
2) can you make the svg option separate, instead of hidden behind the PNG button (with CTRL key)?
3) how did you do the handwriting style?
I once created a dataset of X-hooked-up-with-Y for my circle of friends. About 50 nodes and a few hundred connections (after I asked around a bit for additions).
It became a wedding present for one rather central "node" with the title "drosophila neural regulation network" or something like it. No names, just lines and circles. It's still the centerpiece in his home office.
Autogenerated database documentation is often pretty hit and miss but tbls[1] does a pretty good job in that space. Especially when you comment on your tables, fields, views, functions etc (which is a good habit anyway!) the output is quite useful
I think dagre was ok but oversold the quality of the algos it used a bit. In many cases it can be better to use a SVG file generated from Graphviz directly instead.
I've had good success with cytoscapejs as well, integrating with react so the graph will animate layout on transition. I use elk, which is apparently superior to sugiyama (the algorithm graphviz uses for force-directed dag layouts).
I used to generate most of my diagrams in PlantUML[0], but recently found that my not-taking app, Joplin[1], has Mermaid built in.
PlantUML has the benefit that you can include it in your builds, e.g. parse any .plantuml files in `make doc` or in your CI.
But I like Mermaid more: its easier to read, less quirky and easily integratable in the markdown (or note-taking apps) without the need for extra build steps.
Edit: I don't know of a really good visual tool as I've found text editing .dot files directly with a live visualization pretty nice. I use graphviz pretty regularly.
Graphviz is one of my favorite tools. I use it for most of my diagraming https://github.com/dylanowen/mdbook-graphviz and I use it as debugging output when I'm working through graph problems.
Likewise! I have used it to easily visualize dependency graphs at multiple employers. The pictures can get a bit messy when they're very large, but they were amazing for detecting incorrect/missing dependencies.
Graphviz is an amazing tool, it has just suffered from being cloistered to technologists whose jobs aren't to solve the kinds of problems it is specifically adapted for. I use graphs in security and privacy quite a bit, and even built the tech for a hopeful security platform using a graph back end. They yield the fastest path through complex problems, and I use them to do in a couple of hours what typically would take client staff months.
They're really compelling but oddly unpopular.
Imo, they're useful for one-off pattern discovery, and they're most valuable for finding single or a few exceptions and outliers in normalized data, and you need to be in an environment where there are asymmetric returns on finding those. Surveillance seems to be their default use case, with open ended scientific research a close second. This comes up with graph based recommendation engines, which are essentially a surveillance/marketing product based on preferences. Most businesses aren't based on discovery of anything other than customers for a transaction they already have.
These aren't problems engineers typically solve, which are more about scaling and optimizing, they're more marketing and sales problems, where you're looking for exceptions and opportunities. (security and privacy are the complementary antithesis of these) A graph based product (imo) is ideal for product marketing analysts optimizing for customer preferences and discovery.
From a product perspective, graphs are analogous to ML, where you'd use a clustering algorithm on loosely structured data to yield categories, comparisons, and implied relationships, whereas a graph yields the same thing over the structured normalized data you can feed into it once you have imagined an ontology to fit it.
That sounds really interesting! Can you give a few more details how you use Graphviz? How does it give such a great advantage "to do in a couple of hours what typically would take client staff months."
Did I understand it correctly that you use it to discover patterns? Are these patterns discovered by just using the layout engines? Arent' other tools, e.g. networkx in Python or cytoscape in javascript easier to use interactively in a REPL? What is the typical workflow (maybe plot, find interesting pattern, change query/data in a loop)?
I'm really interested in how Graphviz can be so great. I am currently working with the other mentioned tools for visualization purposes.
Really simply, I typically use Neo4j, but if I have flat homegenous data, I just use Graphviz because the dot markup format lends itself to parsing easily in awk command lines. The times I have used networkx was when I needed a graph abstraction layer to reason about another graph query, so networkx wasn't used as a persistent graph store, but more of an intermediate data structure for orchestrating multiple service and api calls, like a low rent graphql. I'm a crap developer, but the graphs were what i needed to piece the logic together coherently.
One example of clients taking months is mapping counterparties to agreements. Let's say you have inherited a division that has file share full of contracts and you want to understand the line of business. You get the counterparties out of the contracts and find all the paths for obligations between entities within the division and their counterparties. The graphviz/dot layout gives you a map of all those parties in a single slide and shows clusters, instead of a 3 lb. document with a paragraph for each of them that would have cost a massive amount of consulting time, or interviewing several people to get their narrative understanding of how the business worked, the graph provides an objective map.
You could just use D3js, but for me the dot markup was faster on the command line than structuring json.
The idea is if you can formulate a conceptual, narrative ontology of an organization, you can create a grammar of things and relationships, and then you can plug data (contract counterparties) into that model and form a fairly complete map.
Another recent use case was enterprise vulnerability scan data over a very full /16 address block, allocated across multiple divisions under different management hierarchies with thousands of hosts. By linking the host ownership data to projects and an org chart with the types of vulnerabilities, I could demonstrate in a couple of slides what the highest impact patching strategy would be. Again, graphviz for sketching up the ontology, then Neo to do the lifting.
On a much simpler scale that was more graphviz/dot oriented, I did some work for a startup where I worked with the executive team who had acquired a codebase and talent, and created an ontology of their pipeline customers, their stated needs, implied product features, platform dependencies, our service interfaces, their code bases, and demonstrated the flow of how work on the code bases flowed through to impact revenue. This ultimately got represented as a Sankey diagram, but it was graphviz/dot I used to sketch up the initial ontology.
Have you tried gephi? It's not exactly an alternative to graphviz (eg. you can't cluster nodes) but it handles much larger graphs and has a bit more flexibility in layout. It has plugins for both dot and neo4j input.
Wow that looks cool. It reminds me of what Orange.app was for regular data viz, this Gephi is for graphs.
It's pretty notable that the coolest data viz is for discovery, whereas most managers just need to know whether the line has gone up or down. I'm thinking there may be a fundamentally different cognitive orientation to whether one is hunting for opportunity or managing a resource.
I found the learning curve to be quite steep with few examples of graphs that looked well-designed. By default, large graphs look like crap because engineers designed it, not designers (it takes both!).
However, that was back in the late 90's. Now the internet has many examples of better looking layouts, but it is still disappointing that they don't look ... designed?
But integration and automation are great. Like GNUPlot, which I've used for decades because it is so easy to automate once you learn how to make plots look better than the default.
GStreamer uses DOT output by default. But any reasonably sized pipeline is almost impossible to read without excessive zooming. But it gets the job done.
My team uses a succinct graphviz dot file to note the dependencies between different git repositories. This is then used by the build system to figure out what needs to be rebuilt. There are lots of other ways to skin this cat, this is just the method we chose, and it's oddly pleasing.
I've used GraphViz a number of times and highly recommend it as a standard tool on your belt. Having a stand-alone executable that can export to SVG is great.
The most complex thing I've done with it [1]: a tool (MIT-license) that builds diagrams of the data and addressing pipeline for a DSP processor, and lets one 'scrub through' the assembler code frame by frame and see the values propagate through the blocks.
Also PlantUML [2] uses it for most diagrams.
Getting layout and positioning the way you want can be tricky but is usually achievable with patience and hidden objects.
My graphviz-fu got so much better when I started using invisible objects for the grid-like layouts. At that point you're just using graphviz for positioning/spacing/styling instead of creating the overall topology.
if you want X nodes falling near in the graph you just put them into a box with white lines and they move as one from this point. Can be made with the cluster environment.
You're handing graphviz a list of relationships. Sometimes you know the ideal way to visualize this is a tree, or something with right angles, but graphviz isn't laying it out that neatly. So you create extra nodes, and edges to those nodes, so that graphviz considers it a "balanced" tree or grid. Then you use styling to turn those nodes invisible (showing just the nodes/edges you want). Once I have this correct I usually contain it within a "cluster", and then link that cluster to the larger diagram.
Regarding Graphviz itself, I wonder why is there no special layout logic for planar graphs? They can be recognized and embedded on the plane in linear time without intersecting edges, so it would be very nice if some of the Graphviz tools actually did that.
It's great to see it still going. Used it for Puppet as well as dpkg output (we made a full dependency list of our multimedia ingest application, i.e. lots of custom ffmpeg and libboost etc). That thing was huge
I've never been that happy with the output from any of the different graphviz engines. I'd like to see a UI graph editing tool which can import and export .dot format, but also allows manually moving nodes (which I guess wouldn't get exported unless arbitrary metadata is supported), and ofc exports to svg. Maybe there's already something similar I haven't stumbled across yet?
I guess my use case is mainly swimlanes and places where nodes don't have to have the same rank.
That is a different point than "basically abandoned" :-) I do use X11 programs on Unix and MacOS (so far XQuartz works well enough). And probably Microsoft Windows though I don't use it.
I maintain an open-source project [1] that uses graphs to model data. I wanted to make my project as accessible as possible, so Graphviz was perfect since it's dead-simple to install and use on all major OS platforms.
The main downside for me is that sometimes it gets the positioning wrong, and you can see how it can be easily fixed, but it's hard to convince graphviz to actually do so. Basically I'd love a tool where I can do 10% of positioning manually and let the rest be constraint based like in graphviz.
It used to have a built in tool to move nodes it has placed. All I can find scanning the docs right now is the -n flag to neato, which will honor existing pos fields.
156 comments
[ 4.4 ms ] story [ 243 ms ] threadhttps://openprocessing.org/user/15252?view=sketches
Basically any sketch that has "3d" or "aspekt" in the title is on that track. But you can also see from the thumbnail many of them are 3d world related. The navigation of that site is pretty tricky...but you can find the source code by:
- Click on a sketch
- Click on the 'abacus' control menu to the right
- Click on the files tab in that
Source code in Java should be available there.
Please remember it requires a version of Processing in Java to work. I worked on this in 2011/2012 so you may need to figure out which Processing version that is.
Also, here's a youtube video of flying around one of my sketches:
https://www.youtube.com/watch?v=lNshrYO7Kf4
and here are some more videos:
https://www.youtube.com/channel/UCw-6wNeif6-l7oqippdORcw
1. Draw "sketch" style graphs using https://sketchviz.com/new
2. Draw graphs of memory usage in golang programs (using pprof): https://graphviz.org/Gallery/directed/pprof.html
3. Embed graphviz into Sphinx docs (shameless self-promotion: https://docs.robusta.dev/master/catalog/triggers/index.html#...)
I'm amazed that this hasn't been addressed. It's been there for years.
Makes the end products impracticable for distribution in a professional setting. It's Ok if you're the only consumer ... not great for other cases.
If nothing like that exists, I can see how it would be a very troublesome refactor.
The relevant code starts around https://gitlab.com/graphviz/graphviz/-/blob/main/lib/common/... If this isn't right, maybe somebody can figure out why. Perhaps the loop stops sometimes on the wrong side of the boundary? Anyway, it's equally possible that when the endpoint coord is handed off to a lower level driver, the arrowhead mitering is wrong. There is no question there was once upon time explicit code to try to cope with this problem, at least in the native PSgen, but I can't find it now. Maybe it wore away as waves of open source development washed over it, along with static tables for a bunch of "standard" PS fonts.
For more background, see for example https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/s...
In the worst case, this has to be debugged for all the drivers in https://gitlab.com/graphviz/graphviz/-/tree/main/plugin
No doubt there are more details if anyone can remember them.
I'm talking by heart and I could be wrong in any case.
Super intuitive, I know! :)
1) is there an option to make the graph horizontal instead of vertical? 2) can you make the svg option separate, instead of hidden behind the PNG button (with CTRL key)? 3) how did you do the handwriting style?
See in the docs: https://graphviz.org/docs/attrs/rankdir/
In the sketch example:
It became a wedding present for one rather central "node" with the title "drosophila neural regulation network" or something like it. No names, just lines and circles. It's still the centerpiece in his home office.
https://github.com/enter-haken/schema
https://github.com/enter-haken/schema/blob/master/schema.sh
It's a mixture of sed, awk and sql.
[1] https://github.com/schemaspy/schemaspy
[1] https://github.com/k1LoW/tbls
https://graphviz.readthedocs.io
- Viz.js (via Emscripten): https://github.com/mdaines/viz.js; Online demo: http://viz-js.com/
Edit: GraphQL Voyager has the memory increase hack https://github.com/APIs-guru/graphql-voyager/blob/54a178142a...
I've had good success with cytoscapejs as well, integrating with react so the graph will animate layout on transition. I use elk, which is apparently superior to sugiyama (the algorithm graphviz uses for force-directed dag layouts).
PlantUML has the benefit that you can include it in your builds, e.g. parse any .plantuml files in `make doc` or in your CI.
But I like Mermaid more: its easier to read, less quirky and easily integratable in the markdown (or note-taking apps) without the need for extra build steps.
[0] https://plantuml.com/ [1] https://joplinapp.org/
That's how I recently found that Mermaid can't handle graphs with no flow https://github.com/mermaid-js/mermaid/issues/1227
[1] https://wiki.dendron.so/notes/ba97866b-889f-4ac6-86e7-bb2d97...
https://js.cytoscape.org/
https://github.com/jamesshore/object_playground/blob/master/...
https://nethackwiki.com/wiki/Variant
.XML -> Python script -> .DOT -> Graphviz -> *.SVG
It's very useful and probably the best example of using DOT language, unfortunately.
https://www.graphviz.org/pdf/dottyguide.pdf
Edit: I don't know of a really good visual tool as I've found text editing .dot files directly with a live visualization pretty nice. I use graphviz pretty regularly.
Because a lot of big companies use it (and they spend 0 €/$/... in the project), and it is broken from several years...
We do voluntarily act on CERT advisories.
But, true.
They're really compelling but oddly unpopular.
Imo, they're useful for one-off pattern discovery, and they're most valuable for finding single or a few exceptions and outliers in normalized data, and you need to be in an environment where there are asymmetric returns on finding those. Surveillance seems to be their default use case, with open ended scientific research a close second. This comes up with graph based recommendation engines, which are essentially a surveillance/marketing product based on preferences. Most businesses aren't based on discovery of anything other than customers for a transaction they already have.
These aren't problems engineers typically solve, which are more about scaling and optimizing, they're more marketing and sales problems, where you're looking for exceptions and opportunities. (security and privacy are the complementary antithesis of these) A graph based product (imo) is ideal for product marketing analysts optimizing for customer preferences and discovery.
From a product perspective, graphs are analogous to ML, where you'd use a clustering algorithm on loosely structured data to yield categories, comparisons, and implied relationships, whereas a graph yields the same thing over the structured normalized data you can feed into it once you have imagined an ontology to fit it.
Did I understand it correctly that you use it to discover patterns? Are these patterns discovered by just using the layout engines? Arent' other tools, e.g. networkx in Python or cytoscape in javascript easier to use interactively in a REPL? What is the typical workflow (maybe plot, find interesting pattern, change query/data in a loop)?
I'm really interested in how Graphviz can be so great. I am currently working with the other mentioned tools for visualization purposes.
One example of clients taking months is mapping counterparties to agreements. Let's say you have inherited a division that has file share full of contracts and you want to understand the line of business. You get the counterparties out of the contracts and find all the paths for obligations between entities within the division and their counterparties. The graphviz/dot layout gives you a map of all those parties in a single slide and shows clusters, instead of a 3 lb. document with a paragraph for each of them that would have cost a massive amount of consulting time, or interviewing several people to get their narrative understanding of how the business worked, the graph provides an objective map. You could just use D3js, but for me the dot markup was faster on the command line than structuring json.
The idea is if you can formulate a conceptual, narrative ontology of an organization, you can create a grammar of things and relationships, and then you can plug data (contract counterparties) into that model and form a fairly complete map.
Another recent use case was enterprise vulnerability scan data over a very full /16 address block, allocated across multiple divisions under different management hierarchies with thousands of hosts. By linking the host ownership data to projects and an org chart with the types of vulnerabilities, I could demonstrate in a couple of slides what the highest impact patching strategy would be. Again, graphviz for sketching up the ontology, then Neo to do the lifting.
On a much simpler scale that was more graphviz/dot oriented, I did some work for a startup where I worked with the executive team who had acquired a codebase and talent, and created an ontology of their pipeline customers, their stated needs, implied product features, platform dependencies, our service interfaces, their code bases, and demonstrated the flow of how work on the code bases flowed through to impact revenue. This ultimately got represented as a Sankey diagram, but it was graphviz/dot I used to sketch up the initial ontology.
Graphs and infosec go very well together.
It's pretty notable that the coolest data viz is for discovery, whereas most managers just need to know whether the line has gone up or down. I'm thinking there may be a fundamentally different cognitive orientation to whether one is hunting for opportunity or managing a resource.
However, that was back in the late 90's. Now the internet has many examples of better looking layouts, but it is still disappointing that they don't look ... designed?
But integration and automation are great. Like GNUPlot, which I've used for decades because it is so easy to automate once you learn how to make plots look better than the default.
GStreamer uses DOT output by default. But any reasonably sized pipeline is almost impossible to read without excessive zooming. But it gets the job done.
The most complex thing I've done with it [1]: a tool (MIT-license) that builds diagrams of the data and addressing pipeline for a DSP processor, and lets one 'scrub through' the assembler code frame by frame and see the values propagate through the blocks.
Also PlantUML [2] uses it for most diagrams.
Getting layout and positioning the way you want can be tricky but is usually achievable with patience and hidden objects.
[1] https://github.com/paphillips/DFB [2] https://plantuml.com/graphviz-dot
Regarding Graphviz itself, I wonder why is there no special layout logic for planar graphs? They can be recognized and embedded on the plane in linear time without intersecting edges, so it would be very nice if some of the Graphviz tools actually did that.
A recent set of minimal graph coloring Graphviz visualizations of mine: https://gitlab.com/nsajko/example_optimally_colored_graphs
I guess my use case is mainly swimlanes and places where nodes don't have to have the same rank.
My feeling is likely impacted as much from the fact that I've never met anyone that used it.
I get that polymorphism can mess that up, so maybe just link callers to the base class in those cases.
Exports to SVG, PNG, JPG
[1] https://github.com/nccgroup/PMapper
Here are a couple of my diagrams:
https://beepb00p.xyz/blog-graph.html -- graph of my blog pages with tags/connections between posts, generated with a DSL-ish python script https://github.com/karlicoss/beepb00p/blob/master/misc/index...
https://beepb00p.xyz/myinfra.html -- map of my personal data & infrastructure (discussed a year ago https://news.ycombinator.com/item?id=26269832 ). Also a similar DSL https://github.com/karlicoss/myinfra/blob/master/generate.py
The main downside for me is that sometimes it gets the positioning wrong, and you can see how it can be easily fixed, but it's hard to convince graphviz to actually do so. Basically I'd love a tool where I can do 10% of positioning manually and let the rest be constraint based like in graphviz.
Edit: dotty is what I was trying to remember.