HN Explorer

Ask HN: How would you spatialize higher dimensional data?

8 points by anigbrowl ↗ HN
I wrangle a lot of social media datasets, which could be anything from an email database to an old 4chan archive. I'm primarily interested in the metadata network dynamics rather than the content (although I might mine it for hashtags of length/complexity, but I am not very interested in NLP). Sometimes it's just a few thousand items, sometimes orders of magnitude larger.

Typically when I represent this data visually I use force-directed layouts or tree/radial ones if it's definitely hierarchical. And I'm pretty good at the visualization side - filtering, community detection, backbone extraction, layer decomposition, motif identification, time windowing and so on. I enjoy starting out with a giant hairball and translating it into something intuitively comprehensible and explicable. I also experiment a lot with other dataviz paradigms - treemaps, hyperbolic geometry and so on.

But one thing that bothers me about many layout algorithms is that they're so arbitrary; once you go past a certain level of complexity you can get quite different outcomes from the same dataset and starting procedures. And it's difficult to represent agents and their activity on the same graph, eg influencers and their tweets. I am constantly trying to think of ways to map the raw data to Cartesian space or on the interior of a sphere, not unlike a planetarium. But I have some kind of cognitive or knowledge-based block to understanding this, which I can't seem to get around.

How would you spatialize a complex network - eg suppose you had unlimited access to the HN API and plugged it into a particle system or an FPS game engine?

I'm more interested in creating a vivid, explorable impression than scientific modeling. I just struggle with the concept of locality because (in this example) any HN user could reply to or vote/flag any other as there's no 'travel time' between threads or users. And without a coherent spatial metaphor, I can't think of good mappings to 2 or 3 dimensions of space and another of time.

Thanks for your help!

18 comments

[ 3.2 ms ] story [ 63.5 ms ] thread
gus_massa
Locality means make a lot of comments in the same threads. Note that it will make close contacts people that agree a lot and also people that disagree a lot.

I'd try using the 2nd and 3rd eigenvector of the Laplacian of gtaph. I heard a lot of good properties of them, but I never used them so I'm not sure how tricky is to get a nice graph. Something like https://math.stackexchange.com/questions/3853424/what-does-t...

clusterhacks
This is a super-interesting question.

I don't think it is quite the answer to your question, but I have seen combined heatmaps/dendrograms that feel like a possibility for this? Maybe the heatmap deals with activity between nodes and the dendrogram handles the community cluster relationships? I see them often in bioinformatics

Ex: https://en.wikipedia.org/wiki/Dendrogram#/media/File:Heatmap...

sargstuff
Gephi [1] is a software suite to visualize social media datasets.

Demo / feature pages [2][3] show different approaches to visualizing different social media datasets.

Uncertain about current status of software support/development.

-----------------------

[1] https://en.wikipedia.org/wiki/Gephi

[2] http://www.martingrandjean.ch/gephi-introduction/

[3] Gephi home page : https://github.com/gephi/gephi/releases

anigbrowl
I know, I've been using it for a decade. Also a fan of Cytoscape and Socviz, as well as a bunch of more obscure tools in this area.

I should have mentioned my tools, thanks for reminding me.

sargstuff
On the 'more animated' side of things : https://hypertools.readthedocs.io/en/latest/

**

different visual demonstrations : https://www.datavis.ca/gallery/delights.php

Edward Tufte has looked at various ways to present data : https://www.amazon.com/Edward-R-Tufte/e/B000APET3Y/ref=aufs_... / https://www.edwardtufte.com/tufte/

**

Make the quantum leap and use fennman diagrams to construct the network ( https://www.ias.edu/ideas/2009/arkani-hamed-oconnell-feynman... )

**

So, could take the idea of scatter plot matrix in https://www.cs.uic.edu/~wilkinson/Publications/sorting.pdf

and use the various plots to literally build a picture as represented in Excel cells as "picture" : https://www.youtube.com/watch?v=UBX2QQHlQ_I

and

https://jpg.space/mmmatto/exhibition/Mathcastles-%3E-Sandcas...

Hypercastle explorer for terraforms : https://www.youtube.com/watch?v=1jD6F_6_Yak

Although, that kind of starts to get into coding. ;)

https://direct.mit.edu/leon/article-abstract/48/4/375/45993/...

http://lightpattern.info/

https://esolangs.org/wiki/ObjectArt

https://www.dangermouse.net/esoteric/piet.html

**

Obligitory HN 'lisp' link : https://web.archive.org/web/20110504131632/http://sas.uwater...

anigbrowl
Some of these definitely look intriguing. There's a lot of good techniques that have fallen by the wayside for years because they required way more computing power than was practical at the time they were prototyped.

https://treevis.net/ has an especially comprehensive collection of network visualization tropes. I'm not averse to coding and do some in this area already, though in the Lego-assembly fashion of mixing and matching things in Jupyter notebooks with Python or Julia rather than product engineering.

sargstuff
Kinda got the impression poster was trying to think beyond x > log(1)

With a compeling enough story / statics / and lucky enough to land an intership at Industrial Light and Magic, could use the virtual set room used to produce things like 'The Mandalorian' to do a big statimation motion picture. In keeping with the original post discussion, need to use Go programming language; include a van as prop (shoes or actual) as ad lib to getting swept up in ascii screen animations. A few things have to be out of character. ( https://www.vangoghgallery.com/painting/starry-night.html )

treeviw.net ideas mixed with 'massaged statistical graphs' from scatter plot matrix / excel / as 'flip disk' take on https://www.davidhuska.com/~grant/asciimation.html & https://experiments.withgoogle.com/asciimations

Although, massaging the data to fix what trying to present does seem to defeat the whole purpose of finding statistics data to use.

Risks looking like repackaging 'the maxtrix' gimics, even though asciimation was way before 'the matrix'.

Guess the 'ACE' thesis would be creating code structure representing the same visuals as when code file viewed/edited as output of running code to display movie'.

Would really be pushing 'curses' to the limit (sensored or unsensored)

There should be a frame of reference for extracting Lisp from the cons of ascii-izing 'The Cars' with other ascimation movies.

sargstuff
Plenty of visual tropes from various software example galleries.

If weren't for the 2d/3d limitation, would have gone with https://flowingdata.com/2019/10/08/visualization-gifs-imagem... instead of the stata-mation stuff.

* Chart types :

   * latex-cookbook.net : https://latex-cookbook.net/

   * ggplot2 gallery :  https://exts.ggplot2.tidyverse.org/gallery/  https://plotnine.readthedocs.io/en/stable/gallery.html
  
   * "ggplot2: Elegant graphics for data analysis" : https://ggplot2-book.org/

   * Rstudio gallery : https://community.rstudio.com/c/table-gallery/64

   * https://flowingdata.com/chart-types/ && https://flowingdata.com/category/tutorials/

   * https://reference.wolfram.com/language/guide/StatisticalVisualization.html
** xkcd like stuff:

  ** food for thought : https://flowingdata.com/2016/12/12/calories-in-fast-food-menu-items/

  ** heat map in excell : https://flowingdata.com/2020/09/29/how-to-make-a-heatmap-in-excel/

  ** juggling the statistics : https://flowingdata.com/2015/12/15/a-day-in-the-life-of-americans/

  ** https://flowingdata.com/category/projects/data-underload/
** xkcd NLP approach :

  ** https://flowingdata.com/famous-movie-quotes-as-charts/

  ** https://flowingdata.com/2010/05/26/famous-science-fiction-quotes-in-graph-form/

  ** http://www.visguy.com/2010/05/27/sci-fi-movie-quotes-visualized/
** General composibility techniques with examples:

   ** "The Grammar of Graphics" by Leland Wilkinson

   ** R for Data Science : https://r4ds.had.co.nz/data-visualisation.html

   ** "Visualize This: The FlowingData Guide to Design, Visualization, and Statistics" : https://book.flowingdata.com/

   ** Flowing data : https://flowingdata.com/2020/09/08/telling-stories-in-visual-data-driven-essays/
anigbrowl
Many great new avenues of inquiry here - I thought the thread had died for lack of interest. Thanks to everyone!