What a beautiful way to map out the terrain - I immediately feel inspired to use this as a starting point for hobby-research projects. Only problem is now i'm inclined to check the alternatives to "ego graphs" as a starting point for research!
I have a sneaking suspicion that SEO/keyword mapping with all the resources devoted to that space may have some tools that elaborate on this idea - though im no expert. If anyone knows of useful tools to replicate this in browser I am all ears.
You can call the autocomplete API from JS and then filter and draw the graph. Apparently you don't need an API key and it supports firing requests at the rate of typing.
Really cool idea! I didn't find the chess openings graph particularly useful, but what a fantastic way of finding related media! I'd love to generate clusters for my favorite movies, music, and games and use it for new recommendations.
I really wish there was a way to block certain domains from results - alternativeto, slant, Pinterest, quora and more are just noise and often take up the whole first page of results.
Thank you, this is great. I never want to see results from quora, pinterest, facebook, wikipedia, w3schools, .... and these usually occupy much of the first couple of pages of results.
I feel like this ship has sailed. I tried looking for "JobAdder Pricing" a few days ago, and the first 5 pages of results are literally auto-generated bullshit. Feels for the first time in a while like Google are vulnerable to someone coming along with a less shitty search engine (although DDG was no better in this instance).
I use ddg as my primary search engine, and have been doing so for a few months. Unfortunately, at least every 3rd search, especially tech related, I have to switch to Google.
4chan games them all the time, with google then manually removing things from auto suggest (e.g. there are no auto suggest results for "jews are" or "hitler will")
Google actually uses some version of Latent Semantic Analysis which clusters these results based upon word frequency and position. This creates a word matrix where snippets of text can be clustered into themes and the documents are related by a fuzzy concept (document distance in Nth-dimentional space), not directly with key-words like "X relates to Y".
It's harder to "game" that system, but if you know how it works and you're targeting a concept; it'll just force you to create high-quality content so Google wins anyway.
If you really want to play with this type of thing (unlike the blog-post writer that just piggy-backs off of Google smarts) check out the Natural Language Toolkit:
It lives on as autofill in google sheets: if you write down a set of related terms in a spreadsheet, you can drag from the corner to extend the range using those sets, just like you can extend an arithmetic progression or formula
I've found searching the web for things like "when was the fill handle introduced" or "who invented the fill handle" or "what's the history of the fill handle" gives hopeless results.
When I try this, it just repeats the items in order, e.g. "New York, London, Tokyo" autofills to "New York, London, Tokyo, New York, London, Tokyo,..."
Do you need to do something special to activate intelligent autofill, e.g. adding "Paris" in my example?
I don't believe the graphs that appear here. Trying it myself I obtain a very different thing. For example, from "matlab vs." the second completion is "octave", nowhere to be seen in tfa.
This google "feature" is really annoying. I have a private firefox window cookies specific for google searches to mitigate this effect. Yet it is still there.
While the visualizations are cool, I'm a bit afraid of how much of the displayed word cloud is a visualization of the "bubble" that the author lives in. It would be cool to see how different these graphs are on each of our computers!
It looks to me like that graph was produced using k=3 for the k_edge_subgraphs() pruning step, and likely the suggestions for "octave" only included one or two other ML-related terms (because they are crowded out by musical terms).
(For me the "octave vs" query gives "matlab", "Python", "pitch", "rhythm discord" and "key" as the first 5 suggestions).
How would they, without impairing normal users? You need an API for the public that allows you to get suggestions rather quickly without an API key, for all the users that are not logged into their Google account.
Can someone point me to resources on how to re-implement the "vs" feature?
I feel like this can be done with just GPT-2 instead of using the google search api. For example if you type something like "Instead of using Tensorflow, use ..." into https://talktotransformer.com/ it can babble pretty coherently about related technologies so the data is there.
uses BERT, but GPT2 would work as well.
I am working on making this a python package and a technical article on the method which works really great. One example is that I can use clustering to distinguish out syntactic information and then get a purely semantic ego graph (as used in the paper for roles).
I was just kidding. But you make a good point about the starting point which makes me think of an obvious extension: iterate "vs" operation for every node and keep adding nodes until it stabilizes. The set of dogs is finite so the graph must stabilize at some point. If I do "vs" for pitbull then I get boxer and I see pitbull in the graph.
Neat idea, excellent presentation. What a brilliant write-up, I was thinking "I wonder how this works", and then I got to the detailed explanation of exactly how it works.
I decided to try this with my last name, taking the ego graph concept literally. Along the way I discovered the existence of MMA fighter David Hošek and the Hošek-Wilkie model for sky coloration.
I tried it with my online name, tzs, and everything the vs. trick gave was a currency code. It turns out "TZS" is the symbol for the Tanzanian Shilling.
I had no idea.
I wonder if that is why I get occasional inquiries about buying my domain?
First of all, this is an incredibly cool usage of autocomplete. Super-kudos to the author for finding something so conceptually simple to build, that results in data so rich and useful. (It also makes me miss the old Google Sets even more...)
But the results are so good, it actually gets me wondering if Google's autocomplete results for "vs" are actually just "dumb" statistical text mining from the web, or if Google has special code for when it sees "vs" to look up concepts in its own semantic/knowledge graph and generate the autocomplete out of those?
In other words, are these "ego graphs" distilling what is ultimately textual statistics from the web, or some kind of deep learning model Google has applied over that?
(Separately, I would love if someone could do this for the top million n-grams on the web and build a site out of it... I would visit it SO often.)
This is really useful! I've been exploring some Python data wrangling libraries / Pandas competitors (e.g. modin, Dask, Vaex, RAPIDS) and trying to find a nice way to visualise how they all 'link' together (similar to what Jake does in his "The Python Visualization Landscape" talk[0]).
The linked "Flourish" tool from the article is also really nice. It's a pity it doesn't have an API and is pretty pricey for the premium features, but the default network graph looks nicer than anything else I've seen.
I threw together a Repl here [1] (code at [2]) that lets you put in a keyword and produces CSV output that can be copy-pasted into Flourish (excuse the rough code)
68 comments
[ 3.1 ms ] story [ 140 ms ] threadI have a sneaking suspicion that SEO/keyword mapping with all the resources devoted to that space may have some tools that elaborate on this idea - though im no expert. If anyone knows of useful tools to replicate this in browser I am all ears.
Another HN'er referred to it recently and it has been a godsend.
Great stuff.
google vs. -site:alternativeto.net -site:slant.com -site:pinterest.com -site:quora.com
It's harder to "game" that system, but if you know how it works and you're targeting a concept; it'll just force you to create high-quality content so Google wins anyway.
If you really want to play with this type of thing (unlike the blog-post writer that just piggy-backs off of Google smarts) check out the Natural Language Toolkit:
https://www.nltk.org/
https://www.ghacks.net/wp-content/uploads/2009/08/google_lab...
https://books.google.co.uk/books?id=imoPAQAAMAAJ&q=excel+4.0...
I've found searching the web for things like "when was the fill handle introduced" or "who invented the fill handle" or "what's the history of the fill handle" gives hopeless results.
Do you need to do something special to activate intelligent autofill, e.g. adding "Paris" in my example?
I remember the first time I used that, however -- it seemed like absolute magic.
[1] http://www.evolvingseo.com/2014/05/20/google-sets-retired-fr...
While the visualizations are cool, I'm a bit afraid of how much of the displayed word cloud is a visualization of the "bubble" that the author lives in. It would be cool to see how different these graphs are on each of our computers!
https://www.vice.com/en_us/article/m7jvvp/google-sued-tracki...
(For me the "octave vs" query gives "matlab", "Python", "pitch", "rhythm discord" and "key" as the first 5 suggestions).
Google -> Bing
Microsoft -> Amazon
Twitter -> Trump
Messenger -> Whatsapp
Facebook -> Instagram
Instagram -> Reality
Dreaming -> Awake
Awake -> Asleep
Asleep -> Dead
Dead -> Alive
I feel like this can be done with just GPT-2 instead of using the google search api. For example if you type something like "Instead of using Tensorflow, use ..." into https://talktotransformer.com/ it can babble pretty coherently about related technologies so the data is there.
http://ingomarquart.de/index.php/research/2-the-semantics-of...
uses BERT, but GPT2 would work as well. I am working on making this a python package and a technical article on the method which works really great. One example is that I can use clustering to distinguish out syntactic information and then get a purely semantic ego graph (as used in the paper for roles).
I only have one nitpick. The graph with dogs didn't include boxers (which everyone knows are the coolest dogs).
All the steps are there, just not the packaging.
DuckDuckGo: https://duckduckgo.com/ac/?q=test
Startpage: https://www.startpage.com/do/suggest?limit=10&lang=english&f...
Qwant: https://api.qwant.com/api/suggest?q=test
I had no idea.
I wonder if that is why I get occasional inquiries about buying my domain?
But the results are so good, it actually gets me wondering if Google's autocomplete results for "vs" are actually just "dumb" statistical text mining from the web, or if Google has special code for when it sees "vs" to look up concepts in its own semantic/knowledge graph and generate the autocomplete out of those?
In other words, are these "ego graphs" distilling what is ultimately textual statistics from the web, or some kind of deep learning model Google has applied over that?
(Separately, I would love if someone could do this for the top million n-grams on the web and build a site out of it... I would visit it SO often.)
Don't buy a dog. Adopt one.
The linked "Flourish" tool from the article is also really nice. It's a pity it doesn't have an API and is pretty pricey for the premium features, but the default network graph looks nicer than anything else I've seen.
I threw together a Repl here [1] (code at [2]) that lets you put in a keyword and produces CSV output that can be copy-pasted into Flourish (excuse the rough code)
[0] https://youtu.be/FytuB8nFHPQ?t=262
[1] https://google-vs-graphs.garethdwyer1.repl.run/
[2] https://repl.it/@GarethDwyer1/google-vs-graphs