Poll: What should be done about the endless repetition of stories?

428 points by ColinWright ↗ HN
One thing that bugs me about HN is the apparently endless repetition of stories. When something is interesting it gets taken up by several "sources" and then each of these is dutifully submitted by multiple people.

This has some undesirable consequences. One is that it dilutes the "newest" page. That I don't mind so much. What bothers me more is that otherwise interesting discussion gets split over multiple pages, and the same points get made in each discussion, with some non-overlap.

I sometimes revert to my native "engineer" mode and try to do something to fix this. Usually I put cross-references into one or the other so point people to where the discussion is, or might be. Some people don't like this and down-vote them. Others do like this and up-vote them. most people don't seem to care.

I really don't mind the constant dribble of down-votes that I get for trying to prevent the splitting of discussions, but I do care that I'm not seen to be harming the "community".

Hence this poll.

What, if anything, should be done about the incessant repetition of stories?

PS: If you care enough to vote, please upvote the item so people get a chance to see it. If you think I'm karma-whoring and you want to punish me for that, find some of my comments and downvote them as scapegoats.

155 comments

[ 6.6 ms ] story [ 338 ms ] thread
But how many of these where on the front page for any considerable amount of time?
I have no (easy) way to check that. Two of them got 13 points each, all got at least 4 or 5. Some discussion started, but they didn't get very far.

But that's a specific example from today. Statistically, showing that on a day chosen at random there is at least one example shows that it's very likely to happen most days. Certainly it happens often enough that discussions do get split, and it annoys my sense of rightness, my sense of good design.

As an engineer, it offends my aesthetics.

It significantly hampers my enjoyment of HN, but based on this poll it seems I'm clearly in a small minority. I guess I'll just have to live with it.

One of my foibles, I suppose.

Yes, but that's buried in a long discussion about ideas in general, and here I'm interested in a specific, long-standing problem (as I see it) that I'd like to see addressed.

Alternatively, I'll just stop trying to perform a service by cross-referencing things if people - on the whole - don't care.

I'm just tired of people telling me they value it, then seeing that I get down-voted each time. I'd like a sense of what people really think. A poll is the closest I can get to that.

Yes, but that's buried in a long discussion about ideas in general,

I generally agree with the premise of your original post (and I deal with it by providing links to earlier submissions by hand) but here you provide the rationale for submitting stories more than once.

I'm just tired of people telling me they value it, then seeing that I get down-voted each time.

I find them useful, but I only upvote them if I see that they are below the level of the worthless comments or if I can tell that they have a negative score.

If everyone acts like me, you're more likely to lose points on a cross-referencing comment than to gain them.

You've done a lot of complaining about the HN community. You seem not to trust them to do what is in the best interest of HN. Why let them determine what you post?

    You've done a lot of complaining about the HN community.
    You seem not to trust them to do what is in the best
    interest of HN.
Interesting observation. I hadn't thought about it from that point of view. I think you're right. I think I used to trust the community to do the right thing, and now I don't.

That's a useful comment - thank you for making that suddenly so clear.

Other tech solution: you could cluster posts by content. E.g extract keywords/key phrases and group submissions that have some threshold of over-lapping terms. Of course not trivial, but an interesting project.
I voted for merge.

It would also be nice to have an "alternate links" section on top of the comments page, with possibily of upvoting the alternate submissions.

Even better if an alternate link could replace the main one if it is voted as more relevant (ex main blog post replacing a post merely quoting it)

I also voted for merge. Perhaps when a comment formatted in a particular way recommending a merge receives n number of upvotes that could trigger the merge?

It's difficult because so many blogs and other news outlets in the tech media world travel in flocks. After one does some original reporting the rest jump on the story and (usually) cite the original source. However, the original source for the story doesn't always rule on HN in terms of hosting the main conversation. Sometimes this is because of the timing of the submission and sometimes one of the members of the 'flock' are higher profile and gain more knee-jerk upvotes (TechCrunch comes to mind).

quick note - upon submitting the link, there could be a lookup function on the keywords of the title or linked story title, which shows similar submissions...replicating the Google instant search functionality.

yes, there would be obvious issues with keyword overlap and would not apply for customized titles.

Filter on URL would be the first step, if it is a duplicate URL submitted then point the submitter to that story.

Could also use recently, i.e. last week or 24 hours, to compare stories at submitted URLs such that if someone submits a story on G-man advert from a different site it is found or at least present to the submitter to check if their submission is a duplicate.

URL duplication is already done, but it's especially simple-minded, can't cope with tracking crap on the URL, and will never work for the identical story from different sites.

If the URL is identical it counts as an upvote for the original submission.

Let's say that someone reposts an old Spolsky article that has been around for 3 years and was posted back in the day when it was fresh. Instead of detecting the duplicate and not allowing the repost, I'd like it if HN automatically posted a link to the old discussion as a comment on the new post. Maybe do some sort of filtering on dates, so that the system disallows reposts of recent items as it currently does, but doesn't block posting a 3 year old article again for more discussion. Not everyone was here 3 years ago, and asking someone to search HN for every old article they might be interested in is a pretty crappy interface for discussion.

Edit: Removed dangling "There's value in revisiting discussion" statement.

I agree with that entirely, and have no problem with "classics" being resubmitted, with a pointer back to the previous discussion. I'm more irked by the 5, 10 or 20 times some stories get submitted, like the "G-Man" commercial, and others.
I vote for do nothing. If the story is exactly the same just from a different source, it should be downvoted. Some duplicate stories have a slightly different slant and thus encourage a discussion of the subject from a different perspective. The most obvious example is the AirBnB story from the financial times. There is a lot of overlap in the comments from this submission and previous ones, but there are also different perspectives. I think for stories that are this big its better just to leave it alone instead of attempting to cull the duplicate submissions.

    If the story is exactly the same just from
    a different source, it should be downvoted.
You can't downvote submissions.

    Some duplicate stories have a slightly
    different slant ...
In my experience, most don't.

    I think ... it's better just to leave it alone
    instead of attempting to cull the duplicate
    submissions.
Noted.
I had forgotten that you can't downvote submissions (after 2.5 years I still haven't crossed the downvote enabled threshold). That does change my opinion somewhat. I do remember an alarmingly large number of duplicate G+ stories during its launch.

Given that the main way to manage duplicates (by downvotes) is disabled, I could support some other mechanism. I like the idea mentioned in the comments about grouping similar submissions and have the comments merged.

I agree that dilution of new page isn't a big problem and the crowd aspect should ensure interesting links go up.

The problem to solve is split discussions so I think it would be great if duplicate links would be automatically merged and the discussion page unified.

The more challenging problem to solve is how discussions about stories that are almost identical or link to identical stories could be merged. Could there be an option to view discussions independently or show a page that auto-merges similar discussions?

Option not listed: I just flag anything that appears on the front page if another, better post about the same thing is also on the front page. If lots of people did this, then we would quickly wind up with exactly one post about a topic on the front page, so people's discussion would mostly go onto that post. Problem easily solved.
There is too much discussion that spreads across stories though. The idea of merging sounds, uh, dicey, but it would avoid having fledgling discussions from being lost or preemptively killed.
I think the only reason there's a bunch of discussion that spreads across stories is because there are frequently duplicates that sit on the front page for ages. If they were quickly flagged off the front page, I doubt they would attract a significant portion of comments.

  + If lots of people did this, then we would quickly wind up with exactly
  | one post about a topic on the front page, so people's discussion would
  + mostly go onto that post.
If lots of people did this, then we would quickly wind up with no posts about a topic on the front page, because we often disagree about what constitutes "better."
I flag repeats based on who submitted first. First come, first serve. Every next one I see I flag.
Reducing every angle of a story to a single thread isn't going to help matters.

To take this week's AirBnB PR uproar as an example, many of the front page posts on the issue represent different facets of the story. Sure, only one link to the original blog complaint from the renter is necessary. Subsequent posts highlighting the story's path from HN to TechCrunch to a TechCrunch-hosted official response to the Financial Times and other news outlets are providing useful context here. The story has moved beyond the facts of the initial incident and onwards to its impact on startups, fundraising, disrupting regulated industries, and more.

There's a lot to talk about here and it doesn't all belong on one thread.

I think that's different than what the OP is talking about though. If different stories are providing updates to an earlier story or original information of some other kind that's fine. If there are multiple links from different sources that are duplicating the same basic information then it creates unnecessary redundancy and splinters the discussion.
One of the downsides of simply merging items, with different URLs, is that the different articles might both be worth posting because they cover different aspects of a story. So it'd be great to see multiple URLs on one item.

Also, there'd have to be a pretty clear standard about what constitutes one item. Sure, same URL - same item. Probably different URLs for the same story on the same day should be one item. But in the AirBNB ransacking case, I'd argue there should have been multiple articles: one for the initial "This is what happened" blog post and one for the "Suspect in custody, AirBNB has made changes to their organization" followup. Edited to add: And another for todays posting, noting that AirBNB probably hasn't done enough for the victim.

I assume some people blindly post TechCrunch URLs to try to boost karma - so a karma based solution might work. You post a duplicate URL, it changes your karma by -1 or 0. Not enough of a karma hit to really change the overall karma, but perhaps enough to search for the article first.

If things are on the same topic, but provide separate, unique points of view, they should go into an inner list, like you'd see for multiple google results from the same location.
Idea for the merge mechanism. If a user thinks a link has been submitted before; the user posts a cross reference comment whose syntax must be simple as in: MERGE: news.ycombinator.com/link/to/article

If the merge comment gets enough upvote, then a detection algorithm automatically merges it to the other one.

I left out a couple of details, but this is the simplest way I can think of to automate the merging without reverting to clustering algorithms.

I guess this is equivalent to voting the story down the front page, but it allows for salvaging the discussion in the down voted link.

Avert your eyes.
Maybe that was too blunt.

I've watched these kinds of complaints and calls for correction/improvement/action for some time.

I come to HN late to the party, so I don't miss the old days as much as the first residents. In fact I'm probably one of the people ruining the party, for some of the longer residents.

I come to HN every day, because HN is the most interesting site on the web. I suspect that the earlier residents still come to HN daily or with some regularity, for similar reasons.

While I understand the dismay that earlier residents might have in reaction to the changes, these changes are all but inevitable as a more diverse crowd comes in. This is compounded by HN's relatively poor interface, especially in relation to the current complaint.

I don't mean to put words in pg's mouth. I doubt that there's much real incentive for pg to address all the mechanical issues in HN. He has many other interests, and he probably does well to attend to those before this.

Given the realistic improbability of large interface changes, the only thing left is to restrict membership to invite only. Otherwise you'll experience a slow retreat (as you are) from posts that longer residents like, as HN's low but strong buzz spreads.

Since mechanical and membership (I'm guessing) changes are unlikely, the most practical thing for any individual is to not worry about it.

While I'd like a mechanical solution, I think the best results are probably going to come from human intervention/interaction.
I think the problem is that you can not change your vote.

When duplicate stories appear in the front page then you will probably upvote the first one. Problem is that this may not be the one with the highest score, or with the biggest discussion. When you then realize that there is a better submission of the same story, you can't change your initial vote.

I think that if you could change your vote, the problem would be corrected by the community itself.

Weak supporting evidence: reddit does not appear to have the same problem. (At least not on my front page; YMMV.) Of course, there could be any number of reasons for this.
How about a done-vote button?
I'd like to see a tag mechanism where tags may be attached to a submission and voted on. Then, I'd like to be able to view the home page sorted by tags instead of by submissions.

Hopefully, that'd result in all Airbnb stories grouped together under tags representing different aspects of the story as well as automatically cross-referencing it with other examples of PR failure.

I think tags are a great idea, especially if one could block stories with certain tags (i.e. "I don't want to see anything tagged with Bitcoin or TechCrunch").

On the flip side, if someone really wants to read about Bitcoin, then can look at all stories with that tag.

Checkout forrus.org. I'll send you an invite when you do. You can register with your HN name if that helps.
Done, let's see what you've built.
Sure thing, I'm going to write you an e-mail to give you an idea of what's existing and what we're building in the next month or so as well as an invite.
Do it like stackoverflow. Search submitted urls/articles and show in the article submission interface. Preferably ajax-y so as user types url and/or title it already sees if he is posting something repeated.
I was going to suggest the same thing but I'll add that it might be prudent to start cross-referencing posts based on URLs in posts and comments.
I would be a very happy man if I could filter out stories having Airbnb in the title.
So there are two points here: different articles on one topic may contain different perspectives, so they all should be preserved when merging; on the other hand, it would be better if there was a single place where people would discuss a topic.

Maybe a good solution would be to let people vote for merging several articles on the same topic, and once there's enough votes, merge them in one, say by concatenating both links and discussion trees?

If that's too heavy, maybe just extend each article with links to other discussion pages (say, adding beneath the text "This topic is also discussed here (link)").

    maybe just extend each article with links to other
    discussion pages (say, adding beneath the text
    "This topic is also discussed here (link)")
That's what happens with the "by hand" cross-referencing I already do, and which gets so many down-votes.
Merging discussions is IMO a better way, but it's also trickier.

(I, for one, mostly upvote your cross-links - thanks for keeping the graph connected!)

I'd be happy with a hide button
I am not sure what you mean by "merge" mechanism, but it would be wonderful to have a feature like on StackOverflow, where's before you actually click submit, the app recommends that you review suggested links before posting your own.

Of course, this is going to help people who haven't see the same story repeat, but this isn't going to stop those who repost hot topics just for the sake of gaining karma points.

I'd vote for a style of hand-merge mechanism that would

- Take the title of the post with the most discussion

- Turn it into a link like a poll, where each link has it's own "interior" link

- Throw all of the conversations from the posts and throw it into the main body of the combined post.

Users keep the ability to gain points, the front page stays clean, and no data is lost. The only downside is there's no easy way to flag things as "merges", unless the ability to "flag to merge" is added for people at a certain karma level. Enough flags, it automatically rolls them together.