HN Explorer

Ask News.YC: Proposed experiment to test "The PG bias"

21 points by randomhack ↗ HN
It has been frequently commented by people outside the news.yc community that news.yc is too pandering to PG. To test this, I propose the following experiment which requires PG's co-operation. I propose that PG also post comments / submissions etc under some other username and then compare the average upmods for posts under the "pg" username vs the random anonymous username. This can be started at an unspecified time in the future and run for lets say 1 month. Only PG will know what username he is using other than pg.

I also propose the same for nickb and other prominent users. It may turn out that people upmod stuff from more recognizable names when they might have left it at neutral if some other user had submitted the same content.

Well actually it may already be the case that they are posting under multiple names.

Might be an interesting social experiment.

edit : Or is it already underway? nickb is pg :P?

79 comments

[ 5.5 ms ] story [ 178 ms ] thread
byrneseyeview
It would be a good idea to quickly delete this and email pg directly about it, so few people are aware of the experiment.
randomhack
aah .. too late for it now i guess. should have read your comment earlier.
Hexstream
Ah, I thought security by obscurity was out of fashion by now. I'm admittedly out of touch with the "real" world.
byrneseyeview
I thought good experimental design involved not telling people what you're testing if they have any means of altering the test.
Alex3917
"Might be an interesting social experiment."

It's not an interesting experiment unless you learn something from it. There's virtually nothing that can be learned from this.

Plus, I like being able to follow the comments of pg and others on their threads page.

rms
Agreed -- there probably is something of a pg bias, but it's impossible to measure because his comments and submissions are better than the average anyways.
coderrr
How is it impossible to measure? It has nothing to do with how his content quality ranks compared to other people. Since he is writing both the content under his name and the content under a different name, given that he doesn't know which one he's writing under until after he writes it, the quality should be at the same level (on average).

If the quality is the same but the points are different (between his real acct and anon acct), then you've found a bias.

rms
OK, yes, it is possible as long as the other account never answers meta questions about this site or YC, and all meta-questions are eliminated in the comparison. It's still not worth doing, because there is a pg bias: people follow his comments and submissions and are more likely to vote them up because they are more likely to see them.
coderrr
I agree... no meta questions would be compared.

You can get around the problem that you mention by only comparing votes which are made from the main page and not from a "follow" page.

imgabe
If that's the case, then couldn't we measure the bias by having some other user post under pg's name and see if it gets more or less upvotes than when posted under the user's own name?

Edit: To take this idea a step further, maybe have a random chance of any post being attributed to pg and keep track of which ones are actually pg, to see if there's any difference between them.

brett
Being able to follow pg on his threads page is part of the bias too. He gets more votes because his comments get read by more people.
jrockway
Or maybe he just modifies his karma directly :)
coderrr
This is true. But the experiment could take this into account. It could only count the votes which occur directly from the main page.
aswanson
I agree. It tells us what we already know, a lot of people here have respect for pg and it's reflected in his upmods. He founded the site and has written the essays that have struck some resonance with most of the users and drawn them here. They upmod him a lot. What else would you expect?
amichail
Also interesting would be to see whether users who have had an application rejected/accepted vote up pg as much as people who have not applied yet.
technoguyrob
I admit I am guilty of this. Often I realize it after I clicked vote up, but alas, by then it is too late.
Jesin
Yes, the ability to undo a vote would be useful. Shouldn't be that hard to implement, either, given that the site already keeps track of who has upvoted what.

I have also noticed that pg's comments have unusually high ratings, whether or not the actual content was unusually good.

JacobAldridge
Does anonymity defeat the purpose of social bookmarking (and social networking in general)?

I agree that it would be an interesting social experiment, though I'm not concerned if people are pandering to pg, nickb, or anyone else.

amichail
Submissions could be made anonymous for a few hours. I think you would get more interesting items on the front page.
JacobAldridge
I agree - look past who posted something and judge it on merit.

If the pandering is a problem, you'll still get some of the same sites being voted up (paulgraham.com, techcrunch etc), but some other more interesting ones might break through.

Still, I would hope most of us look past those things anyway.

lacker
Just because people upmod more recognizable names, it doesn't mean they are pandering.

For example, I will be more likely to read a comment if it was written by any user that I recognize. If I'm more likely to read a comment I'm more likely to upvote it as well.

attack
But voting like this is screwing up the entire merit system we have here and enforcing group-think.

It's like movie stars getting elected to political office just because they are more recognizable...

fallentimes
No, it's like movie stars being paid more based on past performance. People go with brand names and things they recognize whether it's clothes, fast food, detergent or people. Always have and always will.
attack
...which is detrimental. They won't if we don't let them.
fallentimes
Actually, most the time it's not detrimental. As the post above mentions, it's a very useful heuristic.
coderrr
This is true. It's a heuristic our brain uses so it doesn't have to reevaluate things every single time we experience them. Whether or not this is good or bad is very debatable and depends on the context.

The heuristic is something we evolved through evolution to help us make choices quickly and survive better. For example we know to just trust whatever the group leader says when there is danger rather than having an open debate every time he speaks because that could get us killed.

In the case of HN, does it matter if we have to fully evaluate the content every time before we vote (w/o knowing who wrote it)? I don't know. I think there are probably cases on HN where that heuristic is still to our benefit. I don't think theres black or white here, the costs on both sides just need to be weighed...

Also I think everyone should realize there is a small parallel with the ad hominem attck which is DH1 from PG's post about how to disagree with people (http://www.paulgraham.com/disagree.html). In this case it's exactly the opposite, vote up because of who it is instead of based on the merit, as opposed to vote down (or attack) based on who it is as instead of based on merit.

Harkins
Easiest way to test this would give the class 'user' to links to users. Folks could tweak their browser's CSS with a.user { display: none; } real easy.
dangoldin
An idea I just thought of would be to keep the poster anonymous until the post achieves a certain rating - after that show who the poster is.

If there actually is favoritism this might solve that issue somewhat and yet give people credit where it's due.

[deleted]
(comment deleted)
llimllib
yes, but sometimes the reputation is valuable. Given a poster's prior probability of intelligent comments, you have more information on which to vote.
attack
You also allow him to relax his quality while enjoying his automatic points. Plus group-think lock-in for free.

I can only reducing this making the overall quality here improve.

icey
Find a psychologist, or even a psychology major with enough classes under their belt, and they can tell you that there is most certainly a bias towards upvoting those with a familiar name.

I don't recall the name of the principle, but doing some googling around about Robert Cialdini's work should get you all the information you need.

attack
It could be made so that a nickname is automatically selected per-page. This way threads he comments in could be followed. A small pool of 5 or so names could be reused so that he couldn't be exposed by finding the user that has just been created.

Example implementation:

     unames=['bob1','bob2','pg']
     lu=len(unames)
     secret_seed=309580435

     from binascii import crc32
     def hash_eq(number):
         return crc32(str(number),secret_seed)

     def get_uname(page_id):
         return unames[hash_eq(page_id)%lu]
A checkbox for manual override could also be added.
jcl
Why stop there? Why not have a week where everyone's user id is displayed as a randomly generated syllables based on the hash of their current user id and the current topic id?
kirubakaran
When browsing through http://news.ycombinator.com/threads?id=pg I notice that his comments won't make sense without knowing that it is pg who is making them.

Most are clarifications as Hacker News Maintainer and Furry Investor. (or Furry Capitalist?)

Knee jerk vote-ups I am guilty of are for: edw519, mixmax, rms, pg and nostrademons. But I'm very sure I'd have voted those comments up even if I didn't know the commenter's id. ID gives more context... it is like I hear it in a particular voice.

mechanical_fish
+1, knee-jerk vote-up. :)

No, actually it's the phrase "it's like I hear it in a particular voice". Precisely correct. Sites like this one aren't collections of anonymous robots talking to other anonymous robots.

attack
Yeah, lets forget careful thinking and just support whoever had a good record previously regardless of what they say today.
kirubakaran
The experience of reading a comment knowing the commenter's id makes up (a little bit at least) for the lack of face2face. For example, based on a previous discussion, I know that mixmax had sex with a porn star :) When I read his comments now, my brain does some signal processing magic using this. Please note that I don't give him a +1 coz of this history. But this knowledge puts warmth in an otherwise cold place (cold = place filled with strangers).
attack
Yes, necessarily biased. Severely in some cases, but there's only one way to prove that...
[deleted]
(comment deleted)
mixmax
I should never have written that - how can anyone here take me seriously now?

:-)

mrtron
Voted up, I agree.

Sorry...what were you saying? Why can't people take you serious? That's ridiculous.

kirubakaran
:-) IMO your words carry more weight because of it... (more knowledge, experience etc)
mechanical_fish
I would hope that, if PG and I wrote exactly the same words about (e.g.) Lisp, his post would be upmodded an order of magnitude higher than mine. PG knows what he's talking about as a Lisp programmer, and we all know that. (The published books, the reputation, the history with Yahoo Store, and the existence of this site are big, big clues.) I don't know much about Lisp beyond SICP, and if I claimed I did you would have no way of knowing whether or not I'm telling the truth.

Context, in the form of background information about the author, makes a post more valuable. That's a no-brainer.

Hexstream
"I don't know much about Lisp beyond SICP, and if I claimed I did you would have no way of knowing whether or not I'm telling the truth."

Well, someone as good as or better than you could tell if you're telling the truth. Of course, unless he himself wanted to know if it's the truth then someone as good as or better than himself would have to validate his validation of your claims. So it's a recursive chain-of-trust problem thing.

PG is not exactly the last word on lisp, for example I think a good implementation of Common Lisp knows more about it than he does ;P

mechanical_fish
I think a good implementation of Common Lisp knows more about it than he does.

Perhaps -- but despite decades of effort and ambition, that implementation of Common Lisp still doesn't write very good essays about itself. ;)

simen
That's an argument from authority, plain and simple. Whether or not you know anything about lisp, your comments about it should be judged on their merits alone. If you truly don't know what you're talking about, it should be apparent to those who do know what they're talking about.

A master and a novice might sometimes come to the same conclusions, but seldom because they followed the same thought process. And I sure hope you aren't upmodding posts that don't share any of the thought process behind what they're saying, the reasons why they conclude what they conclude.

Knowing context might make a post more valuable, in the sense that you understand better how it fits into the wider world, but it won't generally make the content any smarter, more valuable, or more correct.

mechanical_fish
That's an argument from authority, plain and simple.

Have you ever wondered why, thousands of years after the ancient Greek experts on rhetoric noted the existence of the "argument from authority" fallacy, we're still awash in authorities? Are we all hopelessly stupid, that we cannot learn to avoid this mistake?

Perhaps. But another problem is that argument from authority is almost never "plain and simple". There's no bright shining line between an argument from authority and an argument based on an overwhelming accumulation of evidence. At some point it comes down to a judgement call.

Suppose a mythical person named Graeme Paul (let's call him "GP" for short) arrived here on news.YC and submitted a one-sentence essay: "Most of the smart programmers that I've met would rather work in Lisp than in Java." My reaction to this would be straightforward: GP is a troll. A very boring troll.

Now let's suppose that GP's post is a bit longer: It begins with his autobiography, in which he talks about his young days as a programmer, and his first encounter with Lisp, and describes the various things he has built in both Lisp and Java. He then gives a detailed, prioritized breakdown of the design of the Java language, comparing Java's features with analogous features in two or three different varieties of Lisp. During this discussion, it becomes clear that GP understands the technical compromises, that he's seen the insides of many real-world Lisp and Java projects, and that he's a gifted explainer. Finally, GP wraps up by summarizing his personal discussions with the designers of Scheme, Java, Ruby, and C.

GP is no longer a troll. He's now making a serious and detailed argument, with many facets, that goes on for pages and pages. He may still be wrong -- history is replete with examples of obsessed experts who were wrong -- but he's got a much better argument.

Now let's suppose that, instead of typing all that information in one monolithic 193-page post, GP spread it out over several hundred blog posts and news.YC comments, posted over several years. At the end of that time, he comes along with a one-sentence post in some language-war thread: "Most of the smart programmers that I've met would rather work in Lisp than in Java."

Because this single sentence is a summary of a long-standing, well-established series of arguments from GP, I might (in the right context) mod it up. Then, no doubt, I would be accused of "not judging this sentence on its merits alone".

The puzzle question here is: Since GP's huge body of published work arguably establishes him as "an authority", is my deference to GP an "argument from authority"? Well, if I've never read any of GP's work but I upmod him anyway because he's famous, the answer is "yes". If I've read GP's earlier work and know that it backs up his latest claim, the answer is "no". Can you, gentle reader, tell the difference between these two cases? Not unless you know my reputation, and/or ask me to provide up to a dozen pages of rationale and clarification.

And there's a huge grey area: If I've read GP's writings on Lisp, and he comes out with a statement about (e.g.) Ruby, how shall I value his expertise when weighing the credibility of his latest statement? It's a judgement call.

simen
Are we all hopelessly stupid, that we cannot learn to avoid this mistake?

Many of us are, though I wouldn't say it is necessarily only stupid people who do it.

I consider it a good principle that the words in a reply in forums like this stand by themselves. If you or pg or anyone else wants to say something about lisp, you got to back it up. If you have already backed it up in "several hundred blog posts and news.YC comments, posted over several years", it should be easy for you to reference some of this material in support of your views.

You saying that the exact same argument, presented word-for-word the same, is not worth as much as the same thing when it comes out of an authority's keyboard, is an appeal to authority. However, if this other person has already defended this view, all he has to do is reference it to make the post worth more. But then the posts aren't word-for-word identical. Then one contains a reference to a justification and one doesn't.

Your example involves a question of how much to trust a person. This is one of the cases where it matters who's making the comment. It's a straightforward question of whether you want to trust that the person who says "Most of the smart programmers that I've met would rather work in Lisp than in Java" is speaking the truth or not, and it's clear that a habitual liar isn't as trustworthy as someone known for their honesty. It's not even an argument, it's just a truth claim.

However, the moment it becomes an argument (e.g., "lisp is better than java"), trusting it on authority becomes a fallacy. If Paul Graham wrote, "Lisp rocks", his comment is not worth any more than your comment if you wrote the same. But if he wrote, "Lisp rocks, for reasons I have detailed in this essay here: <...>", it would be worth more. But then it would no longer be "exactly the same words".

attack_forgotpw
Heavily skewed voting can be WORSE than having no information at all. Considering that 95% of the commenting here is disagreement, THAT'S a no-brainer.
andrewparker
Even if you implement this experiment, I bet there will still be an inherent bias for PG-related material, most notably his essays.

I really don't have a problem with it... that's what this community is all about. Hacker News is PGs world, and if you don't like it, then go somewhere else.

pg
Hacker News is PGs world, and if you don't like it, then go somewhere else.

I don't think this is true. I certainly don't want it to be true.

skmurphy
Persistent identity is an important building block in community formation. Clay Shirky goes into some detail on this in "A Group is Its Own Worst Enemy" http://www.shirky.com/writings/group_enemy.html

"If you were going to build a piece of social software to support large and long-lived groups, what would you design for? The first thing you would design for is handles the user can invest in. [...] It's pretty widely understood that anonymity doesn't work well in group settings, because "who said what when" is the minimum requirement for having a conversation. What's less well understood is that weak pseudonymity doesn't work well, either. Because I need to associate who's saying something to me now with previous conversations. The world's best reputation management system is right here, in the brain.[...] If you want a good reputation system, just let me remember who you are."

I am not sure what the problem is that you are trying to solve, but full anonymity will undoubtedly cause the HN community much more headache than whatever you believe is afflicting is now. Whether there is a "PG bias" or not, he is accountable for his words because they are associated in a persistent fashion with his login.

One final point, I don't know about a pg bias, but I certainly have a positive nickb bias, because he has submitted so many good articles, I will normally follow whatever he posts even if I am not thrilled about the title. Doesn't mean I vote him up no matter what, but making the submissions anonymous would remove an important signal/attribute.

JacobAldridge
I like how this discussion has evolved away from "an interesting social experiment" to a look at how to best promote quality and diversity on HN.

I don't think the original question was identifying a problem and trying to solve it, but the points made here all seek to improve what's working already today.

coderrr
I think you raise an interesting point. There are actually TWO problems here. One is that there is possibly a bias based on the content submitter/creator.

But the other is that, people who are followed will naturally get more votes, just because they get more views. Is that a bad thing? I dunno. But what if it got to a point where there were enough different people being followed that the only way onto the first page is by being one of them. That kindof sounds like a problem too.

JacobAldridge
I understand that some of Digg's changes relate to this second concern. Their issues (I'll admit to limited understanding, so feel free to correct me somebody) relate to their rating algorithm being linked to how 'popular' a contributer is. This naturally weights power users more and pushes their contributions to the front, where they get more views, more votes, and therefore more power.

My (similarly limited) understanding of the HN ranking system is that it's more a factor of time and votes - ie, the question of who contributed is not relevant to the ranking system, only to all of us users who decide which articles to read and vote on, which brings us back to the original question.

zkinion
all the accounts here are bots. Theres like 3-4 real people total. ;)
bootload
"... Only PG will know what username he is using other than pg. ..."

Unfortunately I suspect more than a few will recognise his fist. Just like old school morse signallers can recognise individuals by listening. I'm pretty sure individual writing styles are recognisable.

My idea would be to make user names invisible on submissions to individuals, until after you vote. Only then revealing who they are to the person who votes.

pg
Here are the users with the top median comment scores:

  ("pg" 4) 
  ("theoneill" 3) 
  ("mechanical_fish" 3) 
  ("garbowza" 3) 
  ("kkim" 3) 
  ("paul" 3) 
  ("cperciva" 3)
If there is a difference, it's not huge.
JacobAldridge
Given that karma is cumulative, median is a better measurement. I couldn't spot a standard deviation if it bit me these days, but I would guess the difference between 4 and 3 is significant.

Having said that, a community attracts people like the people it already contains. The more it grows, the greater variety of interests it contains, and therefore the greater number of people with certain, but now differing, interests it attracts.

If you think of it like ripples on a pond, if I'm way out on one edge, I'd likely be vastly different to a user on another edge, though we'd both find some common ground toward the centre. The closer you are to the centre the more you have in common to a greater number of people.

PG is the centre of the pond, so you would expect more people to be more 'like' him, which would lead to more votes. Not bias. Just community.

attack_forgotpw
You fail at statistical analysis.
brlewis
If you think average would be a better measure than median, say why. You might possibly be right, but given PG's track record re. Bayesian filtering, "You fail at statistical analysis" just doesn't seem like the right response. It also sounds like attacking the person rather than the statement.
[deleted]
(comment deleted)
[dead] attack_forgotpw
You are trolling. Clearly you are just trying to get a rise. Regardless, my response:

you think average would be a better measure

I never said that and no I don't.

given PG's track record

Blind acceptance of whatever you idle says. Pathetic, really.

attacking the person rather than the statement

It is you supporting the person instead of what he is saying. I was only attacking the statement that having the highest median score disproves that there is any bias.

[deleted]
(comment deleted)
mrtron
The other side of the coin would be interesting to see too. Is it difficult to determine the number of +/- votes, and could this be displayed on the leaderboard?

It would be interesting to see who is controversial, who makes very passive comments that are always agreed with, etc.

coderrr
I'm surprised no one has mentioned this yet. If you are going to do this experiment, it needs to be _double-blind_. Meaning PG cannot know which username his post is going to show up under until after he has written it. Same goes for all the comments he writes (if you're going to measure those too).

If PG knows before hand what name his content is going to show up under there is a high likelihood of a bias on his part which he would be _unable_ to ignore even if he consciously tried. This is standard psych experiment stuff.

attack_forgotpw
This is easy to protect against. Just obscure pg's real name from a portion of the YC userbase.

This is really an extremely easy test to get right if we think for a minute.

coderrr
That's a great idea. I think it would probably work, but I can think of one potential case where it would introduce some complications.

If user1 sees 'pg' as the submitter and user2 sees 'blah' user1 might comment something which gives away that it is actually pg to user2. Now user2 might realize this and act on their pg bias.

Can you think of a way to prevent this?

kirubakaran
Way:

group1 that contains user1 should be isolated from group2 that contains user2. They can't see each other's replies.

But I wouldn't want such things to happen here except on Apr 1.

mcxx
Hey, who the hell cares? I come here because of good articles and intelligent community, that's all that interests me.
andreyf
> It may turn out that people upmod stuff from more recognizable names when they might have left it at neutral if some other user had submitted the same content.

I think when it comes to social interaction, the ability to build relationships of trust is a feature, not a bug.

jkush
I'm not entirely sure what problem you're trying to solve. Are you trying to "fix" the impression that PG-pandering is present so more outsiders make their way into the community?

I don't particularly care if PG gets tons of karma. I love the submissions, comments and community here.

I think it's best to just let it be exactly the way it is, even if there's some pandering.

bretthoerner
I think the PG bias was proven when a picture of "asterisk man" made it to the top of _Hacker_ News...
TrevorJ
Ok, guys. isn't it working as intended? People build a reputation over time and are judged according to it. It's nothing untoward or unexpected.
jraines
Posts & discussions like this make me cringe a lot worse than actual praise of pg.