224 comments

[ 2.9 ms ] story [ 345 ms ] thread
As Wilson and I are working on this project, we'd love to hear your thoughts!
The visualization of the space is excellent.
It's very fun to play around with, there seems to be value somewhere.

I wonder if you could make a video of how an individual could use it effectively in a pragmatic sense e.g. networking, research

(after more playing around I get the UX, very powerful indeed)

(feedback: add some controls for filtering (dates etc), relevancy, color islands slightly, just general UX improvements and I would probably pay a few dollars a month for this)

(feedback: when looking at a user, you have to zoom really far in before results start appearing. it should probably show labels/results if less than X amount of data points are in your viewport.)

(feedback: Is there "Explore more users" section the more related users to the current profile? It would seem so, but not immediately clear. And if it is, none of my related users have me as a related user in their profile.

===

This is really inspiring work, hats off!

Thanks so much and for the great feedback!
Thanks for making and sharing!

Could you share some info about how you generated the 2D space embedding/visualization?

Thanks! I'll ask Wilson to say more in our next update.
OK, I'm going to just be upfront here and admit that I'm an idiot. This is the third time I've seen this link come up on HN, and each time I've checked it out, but I can't really figure out how to interpret what I'm seeing there in a way that is useful.

How can that map be used to determine who is knowledgeable about what? Looking up myself, I can't connect what I see with my own areas of knowledge.

I think I need an ELI5 for this. Again, this isn't a criticism of the effort at all, it's a public admission of my own ignorance.

Take everything a user says and encode it into some vector. That's "embedding."

Then you encode a search query the same way and return the nearest N vectors/users. That's vector search.

It seems they did the above for HN comments.

Right, I understand all that. I just don't understand how to interpret the resulting map (or how to formulate a search) to provide useful information, specifically "who knows what".
Thank you for being so clear on where we are failing you. We're basically computing your semantic space as compared to the whole community. The map shows the relevant instances in that space. As you browse your concepts the space changes and we link to the relevant comments from the associated threads. Who knows what is prioritizing the user and the semantics / voice of that user amidst the noise of the whole community speaking at once.

We'd love to hear your feedback as you play with it more and we improve all of the above.

It obviously isnt intuitive, but I suppose one could learn to read it over time. For example, there is a lobe at (5, 10) that lights up like a Christmas tree if a user was engaged with Covid vaccines and mandate discussion. Of course, it seems like there is no discrimination between the different stances a user took, or the veracity of their commentary.
> How can that map be used to determine who is knowledgeable about what?

To be fair.. you can’t. Unless you think that mentioning specific words or phrases associated with a specific topic a high number of times means that you’re knowledgeable at it.

Looking at my own profile the words (topics i know about?) seem sort of generic. "issue", "company" - not sure what to make of that other than I assume those are words I use a lot in comments?
We'll take a look at how to do better, thanks!
I feel targeted when you call me a "system server user network number service code issue" ... not that it's inaccurate ;)
Ha, exactly!
Is the knowledge graph weighted more to submissions than comments? I'm surprised to see how generalized the keywords associated with my user are "Company, Product, Market, Part, Article, Issue, Bit".

I'll admit, I am a generalist, and I comment more than I post, but I would have thought/hoped my knowledge base would trend towards neurotech, sleep, mental health, health, wellness, etc, as I feel that is the "less general" stuff I comment on.

Though I recognize this is how I like to see myself, I do often comment on business models, branding, etc etc, which are areas I have interest in.

I wonder if topics that have more activity from a user, but are in a less popular topic area would improve this?

Right now it does end up representing quantity + quality than just quantity alone. How can we do better?
I'm thinking you'd have a huge volume of people with similar common attributes, and if these most common attributes/keywords suggest a more baseline understanding of the community as a whole.

I'm not sure how you surface the more nuanced understanding of each users knowledge or interests.

The few things I'm thinking are 1) more weight to more recent content than earlier. We change and grow as people. I used to be in the music industry, then 3d mapping, and now neurotech. My music experience is now old, but neuro is fresh. That may be more interesting/valuable.

2) commonality with the viewing user. Where do we overlap, and again, where do we overlap where others don't? How often would a person be searching for someone whom they don't have overlap with? We are likely interested in the same things. Though perhaps somebody who knows nothing about marketing, is looking for marketing help, so I'm not sure how you surface that.

I guess I'm wondering what you imagine the ultimate use case being? Or the early adopter use case. You may be prioritizing the algorithm for uses I'm not expecting.

When you click on some keyword it's actually to go back from that keyword.

It would be nice if map showed titles of all articles on extreme close up.

Overall it's interesting tool but I think for people with small amount of comments this is not much informative.

Wow I really like the topographic visualization of the data, looks really slick!
Thanks so much!
This is really cool. Also a healthy reminder that anything we post publicly is likely to be analyzed. With a little analysis it's probably easy to know me better than I know myself.
Maybe also who you'd tend to get along with and your unique place in the community?
Would be interested in a study, unfeasible as it may be, on the social compatibility of people who have the strongest overlap between themselves and the next closest user.

Cool visualization and analysis, really well made!

Thanks so much! How do you see that study playing out in metrics that we can measure and examine?
Email adjacent users with a chat room link, say what this is about (you were identified as being similar, maybe you have similar interests)

Email random users (control group) the same

Ask after a month who's still in touch

I'd participate but please send me some of each group and don't tell me which is which :D

Absolutely no idea :D but if you were able to quantify it somehow I think the results would be fascinating.
For the “terrain contours” is there something specific you’ve done to make it feel more cartographic? Or is it basically just marching cubes / iso lines on some data points?

Looks fantastic. Very cool project.

I'll ask Wilson to chime in on the question; thanks so much!
When I did this, I found it to also be an interesting way to fingerprint users and find alternate accounts. I was able to match an old account I used in a top 10 similarity match out of all users.
HN and other forums perpetuate a fallacy (that I assume isn't well known because nobody has published a research paper on it)... but "the wisdom of the crowd" isn't.

"Trusted voices" on HN are often not experts, yet espouse views which are not what the larger body of experts would consider correct. In addition, actual expert voices are drowned out by whatever the popular position is. HN also provides its own cultural bias, in that everything spoken on HN has to follow a rigorous cultural sieve set down by the guidelines, such that a "negative" view, even if correct, is considered either wrong or distasteful and buried. This is exacerbated by banning the use of humor to disarm controversial or heated comments. And this is the comments that HN does get; many opinions are never entered as comments here, so there exists a large knowledge gap. Then there's the "taboo" subjects like race, gender, religion, politics, social justice, etc which get buried for fear of controversy, so you're definitely not gonna find any expert opinions on those, as the stories just aren't there for discussion.

The end result is that often experts go unheeded or even downvoted, popular shallow opinions get upvoted, and substantive commentary based on evidence and experience is frequently missing. The fact is that we have no idea who knows what, or what's true or right. We just have "popularity" according to the particular cultural quirks of this site. So you can definitely find out "who thinks what", and who is considered to be more trustworthy to a HNer, but it has absolutely nothing to do with objective truth or the body of real knowledge that exists outside the world of HN comments. This is an echo chamber, but it's not a chamber of experts. It just seems that way because occasionally you see a minor tech celebrity, and people talk with absolute authority regardless of if they have any.

You want to find out who knows what? Look at their diplomas and careers. If they've done 20 years in a single field, probably they're an expert. If they have a degree (or multiple) in a field, probably they're an expert. If they spent half their life working on a single hobby, they're at least very knowledgeable in that field. But you can't determine that just by looking at who's talking about what or how many completely subjective "points" they get for what they say. Determining real knowledge requires analysis of specific criteria, filtered to get a higher quality result.

Humor is not banned on Hacker News. Bad attempts at it are penalized, though.
It's not "banned" but 90% frowned upon.

This is a serious site, people!

IMHO if there's any take away from the last 20 years of the internet, it's how hard it is to have decent communities of people online, curated like a blooming garden of diversity. What are some online communities that you appreciate?

I have a Ph.D. in neuroscience with exactly 20 years of experience. I talk about what I know and try to be curious about what I don't know. HN really helps me on both counts. How does it help you?

Personally, I like how HN focuses on content and discussions rather than individual users. If I wanted to follow experts, I'd probably curate a selection on a social network like Mastodon, or kludge together some RSS feeds.

Also, I feel like this tool selects for active commenters, not for knowledgeable experts. Not to mention throwaway accounts.

Still a cool project.

Thanks! Those are fair points. We're thinking we could uplevel the social layer so you can connect with people of similar interests for deeper connections. In this way we compute not just your contributions but how they relate to others.
With such efforts I think it is time for people start deleting their accounts, not because they want/need to hide anything, but some people may want to stay semi-anonymous by using aliases and data mining everything and correlating it with other sources (e.g. LinkedIn) may help identify them and cause trouble (e.g. someone wrote something about their workplace without naming it, but hey, it is "John Doe that works in MegaCorp".
> it is time for people start deleting their accounts

HN does not let you do that though. At least last time I asked about it, they sent me a response saying that they’d notify me when it became possible to delete an account. And they provided some reasons for why they don’t delete comments and accounts.

It would destroy older comment threads. There are quite some gold nuggets to be found there.
Sure, but what is more important? Some random comment threads, or honoring the wishes of the people who have been contributing comments to the site?

For me I would prioritize the wishes of the contributors.

Well, how about the contributers honoring the wishes of the owners of the plattform?

Also my wish as a contributer is that the threads stay like this. So I can come back later and reread them. I often gained value in reading a linked old thread.

My advice for people not wishing for that, would be simply to stop commenting instead of demanding the site should change.

Could there be a middle ground to remove the account ID as a correlation between posts? The posts could remain as a graph of replies without retaining who posted which.

I know this means someone could still use stylometry to try to reaggregate the posts, but that's less reliable than HN actually telling us that the same authenticated user made all these posts over a decade.

I could very much live with that solution.
I thought it goes without saying, that when you post something sensitive, that you don't want to connect openly to your account - you use a throwaway account. HN made it easy to do that.
"Witness Mr. Henry Bemis, a charter member in the fraternity of dreamers. A bookish little man whose passion is the printed page, but who is conspired against by a bank president and a wife and a world full of tongue-cluckers and the unrelenting hands of a clock. But in just a moment, Mr. Bemis will enter a world without bank presidents or wives or clocks or anything else. He'll have a world all to himself... without anyone."

-Twilight Zone Episode: Time Enough at Last-

The social web died, all you're doing it making pitchfork and torch 2.0 for mobs.

If you want to add value and not bloody public spectacle rank comments instead of users.

I have a bunch of low quality posts here when idiots piss me off, but also share world first research and breakthroughs I've been involved with the rare time the counter party is worth talking to.

> rank comments instead of users

> I have a bunch of low quality posts here when idiots piss me off

It’s ironic because in the second of the parts I quoted you on here you are basically yourself generalizing the users instead of the comments.

Is it really “idiots” that piss you off (the users)? Or is it the specific things they said in isolated cases (the comments) that piss you off? Wasn’t your point exactly that this kind of distinction is important?

And the OP here seems to willfully ignore the main point his parent was making. He said that not having this focus on a "social layer" made HN better. But then OP says:

    We're thinking we could uplevel the social layer
head => palm
It’s great that you finished it but I wish you had taken a little more care to protect users here. Some people I respect a lot are a little more vulnerable because of this
This seems like a slippery slope towards bog-standard social media.

Currently, HN is the only place on the internet I am willing to interact with others _because_ it lacks the "social layer" you are recommending.

The focus on user comments that are thoughtful, relevant, and respectful _is_ the social connection I value.

I concur on the sentiment - I think it’s great we don’t have avatars etc here as it makes it more content focused -

But at the same time I think the OP has made an awesome project! Well done

I agree. It's also quite humbling which I think is a good thing.
I think that especially in CS, since applications of which touch on nearly every possible field of knowledge, computer scientists often run into trouble of assuming they know more than we do.
CS people are prone to the engineering trap of "I've learned one slightly complex thing so obviously I'm capable of knowing every complex thing". It tends to forget how important sheer quantity of practice is in every field that expects a higher than high school level of education.
The negative space should be the most interesting, since absence trends are the hardest to recognize. What are Hacker News’s quantifiable blind spots? Answers on a postcard.
I've been interested in a map of what gets flagged for a while.

Not the negative space, but a list of untolerated content.

Personally I don't read user names. Easiest way to focus on the comments. Of course, the lack of avatars and signatures and stuff like that helps a lot.

I have absolutely no idea who I've had an argument with on HN ever. I'm sure I've had a few.

It's interesting to keep track of some known people. Sometimes you get to see (for example) a thread with cperciva, tptacek and animats - and I think it makes it more fun to read when you know...
But but ... they're known to you because you read the user names ...
I’ll often looks at the username if I find a comment either particularly insightful, or particularly stupid. It tends to be the same small collection of users, so I guess I’m agreeing/disagreeing with people quite consistently.
There is one person here that is like my Polish doppelganger. I'll go to post in a thread and I'll find they already posted the gist of what I was going to say.
If I see a thread with too many comments from high volume posters, I assume it's low signal. Most of them are very good at commenting in a style that is convincing sounding and popular here, but when they stray into topics that I know well, I often find them to be confident, plausible, and wrong. (And also upvoted above people with correct information who don't have the same name recognition.)
>I often find them to be confident, plausible, and wrong. (And also upvoted above people with correct information who don't have the same name recognition.)

This is basically 99% of hacker news comments in a nutshell.

You pass the confidence, plausibility, and accuracy challenge with flying colours!
I'm sure that happens with me (I've noted for awhile that banal comments I write here get upvoted, presumably from a combination name recognition and people who engage with HN by following specific people --- which is how I do it too). What I'll say is if you see this happening with me, call me out on it! Refer back to this comment, if you like, when you do.

Apart from sheer volume (I have been a message board nerd since FIDONet when I was a teenager, it's second nature for me) I attribute what success I've had here to engaging principally on security topics, which is where I've spent my whole career. But I also just like to shoot the shit about stuff! I'm wrong all the time! (Including about security stuff).

One username that always stands out is “1970-01-01”. Every time I do a double-take, thinking a timestamp has gone wrong.
the same, I get weirded out sometimes when I read a remark from someone who evidently has decided to keep track of who I am on the site. Seems like a lot of wasted mental effort for little reward.
A couple of weeks ago someone made a whole anonymous Mastodon account just to ask me a vague question about a comment I made here on HN like six years ago. It kinda creeped me out.

A fan? A stalker? Just a rando with too much time on their hands?

I do look at the commenter's name (same as for an article's author) as I know quite a few commenters personally (some are former colleagues) so our replies sometimes refer specifically to something the other person knows or did.

Obviously this applies to a negligible percentage of total commenters, but as I only comment on certain topics I’m more likely to encounter friends with the same interests/experiences.

But curating such a list of experts to follow takes quite some effort. It would be great to have a tool that helps with that.

And sure, ideally you wouldn't need such a list of trusted experts but just focus on content. There even was a time when this worked - you could just type "what is the best database to use" into a search engine, and get a helpful result. Not anymore. On HN it may still be better than elsewhere, but ultimately it's a similar issue.

Hacker News encourages more meaningful and relevant discussions
> "Despite the intervening 16 years, we're amazed that social networks, even Hacker News, don't compute and display the trusted voices across topics. Instead of prioritizing pages based on content, social networks could prioritize the people behind the content."

Fallacy: appeal to authority. Practically, just because someone generates great content on subject A doesn't mean their take on subject B is any better than random. A well-reasoned self-consistent argument informed by accurate data is far more valuble than 'trust this expert opinion because this expert can be trusted' approaches - although it may require more work on the part of the reader. Don't get lazy.

We're trying to stay away from authority and just compute based on what you say then transparently show where that information came from. How do you think we can do better?
https://hn2.wilsonl.in/user/simonw includes "Risk of COVID from pianos" down at the bottom of the map. I'd love to know where that came from!
Great call out! Would it help to make the points more interactive?
Can you please remove the email address from the user profile page?
I wonder if there's a risk of pianos from COVID?

Kind of like how when the pandemic started, sales of guitars and home gym equipment went up as people worked on at-home hobbies.

Mine seems fairly accurate - accelerating RDP, 3d printer firmware, touchscreen Graffiti gestures (which kind of dates me) and mainline Linux in SBCs, all of which are my hobbies (I seldom, if ever, comment on actual work, like cloud architecture, networking or... AI, which is too crowded a field).

Weirdly, almost no mentions of Apple stuff (which is supremely ironic given my blog).

So you get an impression of what people discuss here, but not necessarily what they know :)

Just curious what the mechanism is for adding a bio to individual users. Your sample -- robg -- has the same bio on your stuff as on HN but others have stuff on HN and "No bio provided" on your thing.
Right now if you add it to your HN profile we'll be able to include it.
One data point:

I checked my profile earlier and there was none, though my HN profile has scads of info. That's why I was curious.

Totally fair to say "Word limit!" or something, but it means your reply doesn't seem to jibe with my observation.

Thanks, we’ll take a look!
Some sentiment analysis might help? Apple is one of my keywords, but you'd pretty-much only get an "avoid" response from me (to a degree, I have recommended their phones to a very specific demographic).

Awesome project, fantastic UI.

Same. I think a sentiment analysis could be helpfull.

I find myself dragged to Apple discusions more than I would like to admit, but they are generally negative in the last years.

It's a topic we keep coming back to, seems like we can do better than sentiment analysis to show the semantic space of the sentiment(s) and how those breakdown by users and groups. For instance assuming a large discussion set of apples, different types of apples, and other types of fruits, we'd discover the patterns among users. Have you seen folks doing that type of semantic sentiment analytics?
(comment deleted)
Super cool! Happy to see that I show up under documentation-related stuff, considering how frequently I mention that I'm a technical writer in my comments. Also pretty excited to connect with the other people that show up related to docs / technical writing / etc.
As opposed to me not showing up at all for things I comment on. Guess I'm not knowledgeable enough.
The results are biased towards users with high post count. It doesn’t matter how knowledgeable you are about a topic, only how much you comment on it.
And using the correct keywords. SEO is dead, long live SEO!

Also I imagine that my area is pretty niche (not many people talking about documentation on HN) so I may have less competition

We'll take a look!
How does determine which comments are valuable ("trusted") without access to the vote count of comments?
We're trying to stay away from karma and focus on what people are saying within the community, rank orders, overall semantics, and more. How do you think we can do better?
I'm ignorant on how you are measuring quality, but I think an interesting blend of karma with whether or not there are replies that argue. If the replies are generally positive, that may indicate quality. If the karma is below 1, and the replies include a number of refutations, that can indicate low quality.
I don’t think comment “score” is available in the api.
Correct it's not (at least that I can tell), but there are some (indirect) ways to approximate that number based on order, time, and/or color
Yeah I was thinking this as well, though the indications of color (maybe dead-ness also) would probably have to be scraped from the front-end.
Recent and related:

Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun - https://news.ycombinator.com/item?id=40307519 - May 2024 (159 comments)

Thanks dang! Amazing to me how much HN brought Wilson and I together after 16 years and across the world. We're hopeful about the future of trust on the internet exactly because of communities like HN.
> Mapping Hacker News to find who knows what in the HN community

Would things like this create/increase incentives to game the metrics?

In my limited experience, people tend to gamify any numeric metric readily apparent to them.

Anecdotally, total game time (in hours, usually) is used to convey experience in video games (WoW, CS:GO, PUBG). I've seen people create & run 3rd-party software to artificially inflate these types of metrics.

And some metrics aren't just a "game", but are gamed for very real benefit (e.g., gaining admission to a college, getting a promotion, being more employable) in ways that are counterproductive to pretty much all other goals.

So, when something like HN has goals, and you create, say, something akin to a contest and link it to real-world benefits (e.g., people expect a recruiter or hiring to use that "who knows what" info), then many people will modify their behavior, probably to the detriment of original goals.

We're trying to stay away from karma for that reason and focus on what people are saying within the community to find trusted voices. How can we do better?
How can Web sites do better at producing valuable and trustworthy content, when what everyone cares about now comes down to SEO?
Why? All my game times are inflated because I either don't shut it down when i go away or I alt tab out of it, do something productive and forget about it :)
I hate this. If i used my real name id have some pleb website on the internet making chump guesses at what i know based on what i was prepared to share. Thanks for reminding me to use anonymous burner accounts on everything app i use.
Thanks for the feedback. How can we better support authenticity and trusted voices, where you fit in, and others like you?
Dont bother. Im never going to be your demographic. I don't want to be profiled by anything for any purpose especially when 1. that profile is then shared publicly 2. I never got consent. I don’t need tailored recommendations and im not going to assess people based on a third party’s black box interpretation that cant possibly have appropriate context.
I'm not quite as extreme as OC, but value privacy enough to not carry a cell phone and always pay cash; my suggestion would be to allow an ability to opt-out, which would unfortunately probably require you to retrain your entire model (i.e. to destroy all connections/weighting to the opt-outs).

An example would be that users place a small opt-out snipped within their user profile, e.g. [no-robots] ?

Perhaps OC might consider never commenting outside of 4ch..? Except even glowies train LLMs =D

I share on hacker news because:

1. I want the knowledge/experience/anecdotes/opinions of others. 2. I want to share my own knowledge/experience/anecdotes/opinions with others. 3. Contributing low-bandwidth, high-quality* content is (hopefully) doing my small part in encouraging others to do the same.

My primary concern: You are providing a map of individual user interests over a period of time. This is feels like a violation of privacy to some and is outright dangerous to others.

Why? 1. People have many interests. The Marketing Manager likes birdwatching, Bollywood, hacking podcasts, amateur chemistry, traditional Ukrainian breadmaking, ASMR, erotic independent film, rock climbing, pole-dancing, gunsmithing, and horse racing. 2. People are public with some interests, and private with others. The degree of public vs. private disclosure can come from external pressure (ie: political oppression, public servant expectations) or internal motivation (ie: embarrassment, pride). 3. People change over time. The 27 year old Marketing Manager may want to distance themselves from their poor political comments as a student.

Where to now? If you stop work now, someone else will pick up where you left off. So how do make a system which balances the user AND allows you to continue?

1. Easy and simple opt-out option. 2. User profiles. Allow me to share /which/ interests are associated with me. I am a nuclear engineer by trade, but I'll be damned if I want to talk about it in my spare time. Let me opt-in to Birdwatching, and opt-out of nuclear engineering so those weirdos at work don't find me. 3. Limit the Time for association. Maybe that means you only 'connect' people with their comments in the past year as the Public default. Once the user connects with your system and has control over their online persona, then the user can decide what to show and for how long.

Future use? 1. Push this tool as an anonymity improvement tool, which also encourages meaningful conversations with people you want to connect with. 2. Feature: "We notice you posted for the first time to /r/bsdm with your /u/PepsiOfficialMarketing. Was this a mistake?" 3. Feature: "Our AI has scanned your recent posts. We have connected your writing style across two accounts and this may compromise the seperation of accounts. The following phrase: 'Do what thou does MFK!' was used to connect these accounts by our AI."

(*Compared to current internet gasoline fire of ad-infested, algorithm targeted, Top-10 lists of Top Ten Lists.)

Good God I am a bore. May track some people down in my niche interest areas though, thank you for this.
Oi! Watch it! Our distance in embedding space is small. Then again, maybe I am a bore...
I found it a bit challenging to actually drill down into my own username, but it doesn't seem to offer much other than throwing a lot of dots all over the map. I'm trying to understand what the overall clusters might be, but most of them are just android/apple/google?

https://hn2.wilsonl.in/user/mmastrac

So my expertise in apple, google and tesla is not so unique?
At least you have some expertise... Mine comes up blank...
Searched for "Buttplugs".

My name pops up.

Hell yeah. A++++ completely accurate would use again.

Haha, now I must track down the contexts of buttplug conversations on HN.
Interesting. I have noticed that my most-upvoted comments relate to legal questions, which I have relatively more expertise than most HNers (I used to be a lawyer). Although I'm not among the top commenters for law/legal/lawyer according to this tool, I definitely recognize some of the top names and can recall seeing their comments in legal threads. Pretty cool tool!
Does the HN API include comment score?
No.
Wait, then this tool is based on volume of references?!
That would be weird. I rant a lot about spirituality and religion and it doesn't show up. But that one time I talk about pregnant man emoji and now I'm the reference for that.
Very interesting, thanks!
Things I thought I had expertise in and have posted about disproportionately on HN: chess, Python, HFT, Fermi estimation, Ireland.

Keywords the site actually associates with me: language, English, article, team, book. To be fair, at some zoom levels I do get "chess".