Ask HN: Do you think many posts don't get the front page attention they deserve?

77 points by sph ↗ HN
We all know about the tyranny of user-upvoted content. Some good posts fall through the cracks and never get the attention they deserve.

But I think HN is suffering from a slightly different problem. There are so many good posts that often extremely relevant and well received content never even hits my front page, and I am on this site a lot. I often use the "Search" feature to surface articles I am interested about, and many times I happen upon posts in the 100+ votes from the past day or two that somehow I didn't even see.

I wonder if the really quick decaying algorithm paired with the amount of quality posts creates the unfortunate situation that reaching the front page or the intended audience is just a matter of pure luck. Have we reached a singularity point where Hacker News has become too popular and on average too interesting that content relevant to your interests rarely reaches you?

EDIT: ironically, this post is now on the front page. If there's one thing I love about this site, is how it's slightly biased in favour of Ask HN and Show HN posts. Even in such a massive forum, everybody gets their 15 minutes in the spotlight.

113 comments

[ 7.3 ms ] story [ 293 ms ] thread
Yes I think so, the front page is always going to be relatively inane and unchallenging as it represents the curated intersection of the interests of the many, which is often boring and conventional.

I never use the front page because I have little interest in what the collective finds interesting. I only ever check the newest page and I have showdead set to true so I can see things as they come in, and I think that's a far more interesting way of consuming HN, if you can tolerate the occasional spam imploring you to buy earthmoving equipment.

> Yes I think so, the front page is always going to be relatively inane and unchallenging as it represents the curated intersection of the interests of the many, which is often boring and conventional.

To use a movie analogy. The top half of the first page are all blockbuster superhero movies. Everybody likes them, they sell more than any other movie, but it's not exactly the cool niche indie movie you're looking for that drives you to this place every day.

> The top half of the first page are all blockbuster superhero movies. Everybody likes them

I hate blockbuster superhero movies. I must be an outlier, otherwise they wouldn't be making them.

You and me both.
I know this is an unsatisfying answer, but the best solution to this problem is to click on page 2.
Only if the problem is "I didn't click page 2." Personally, I consider the first 2 pages as front page, and check page 3 at least once a day.

/new could be good, but there's far too much spam to go through. Checking out new posts quickly turns you into a janitor where you spend too much time vouching or flagging stuff instead of just looking for something interesting.

This is the problem. As i said in another comment, practically noone bothers visiting /newest.
Yes, I do think lots of post don't get the attention they deserve. That's why I check the front page only once or twice per day, and otherwise live inside the New page.
Probably, i use the Android App "Harmonic" for Hacker News and i can and do scroll down until the 499th post and see many stuff that some people won't see. I often upvote them and sometimes bookmark them but looking at the posts i bookmarked i see that those posts stayed down there. Which is a shame as i really wanted to see the comment sections to these post, thats (IMO) the best part on hacker news, which often stay completely empty.
There should be a tagging system
Great idea, I like the lobste.rs tagging system. How to avoid abuse when you have so many users is a hard problem though IMO.

I really don't want to pile even more work on dang's shoulders :-)

You could allow for community tagging and upvote/downvote on tags, which would reduce the moderation requirement (hopefully).
#Education #Debate #Question #Community-Notice #Hyphens-verusnullspace #InsideJokes/=Inside-Jokes #Entrepreneur-Spotlight #Sponsored #New-Tech #Old-Tech #Complaint #Thread-Update #Another-List
I think I've seen such a system work well in a few instances, but it needs to both incentivize good tagging and disincentivize mis-tagging. The latter needs to be something that actually hurts users, makes them feel the real pain as a consequence of mis-tagging. I'm not sure how to generalize that from the examples I have in mind, which are all... porn drawings sites. Imagine being into a specific fetish, enjoying yourself clicking through new pics, only to get that one mis-tagged pic that should have been tagged with dirty-old-men but wasn't...
That would make moderation even harder. The set of tags needs to be hard coded, and perhaps giving some tagging privileges to high karma users. (I have decently high karma, so I might be biased.)

Upvoting/downvoting tags would encourage memeing and create even more spam. Steam has community tags w/ voting, and sometimes you find very hard games tagged with "Relaxing".

Absolutely. Posting to HN is a bit like shouting into a hurricane. During peak hours, there are often several posts per minute. Even good posts require significant luck to make it to the home page.
A critical variable here is the size of the frontpage.

Perhaps people tend to assume that the size of the front page reflects the volume of posts that deserve attention.

Maybe front page size should be configurable; someone could make a browser extension for that.

I think the number of posts that reach the front page is precisely the same number of posts that deserve to reach the front page, as the means of determining worth is entirely self-referential. Thus the purpose of this system is what it does.
There's actually an external principle we're optimizing for: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor.... On HN, the optimal front page is the most intellectually interesting front page.

That's an oversimplification, of course, because every user has a different fitness function for what's interesting. So the optimal front page would be something like the front page that aggregates everyone's fitness functions in an optimal way, whatever that is.

I'm not saying HN gets there, or close. But it's not for want of trying, and we at least know what we're trying for.

It's a marvel it keeps working as well as it does. Even if we could introduce more external input in the design of the core machinery, I personally would not.
You raise an interesting point, which immediately leads to the question of how it could be improved or solved, without significantly affecting how the front page works currently.

For example, a layer over the basic front page might offer stories thought to be of greater interest to each individual user - essentially, a recommendation algorithm. Maybe submission/commenting patterns (which are basically available [0] via the API) could be harnessed towards this - taking the hypothesis that users interact with stories which most interest them, and there are patterns of similar interest between people?

Or maybe there are other opportunities; for example, if story voting was made available per user via the API, this would add an additional stream that roughly signals "this interests me". (Although of course this miight consequently cause other problems...)

[0] https://hacker-news.firebaseio.com/v0/user/jl.json?print=pre...

And an auto-repost feature. I still don't understand the difference, I think it depends who "rescues" the article (dang or unnamed power users).
First I've heard of auto-repost. Is this documented somewhere? Is there evidence that it exists?
If by auto-repost you mean when we email someone inviting them to repost a story, it's exactly the same thing as the second-chance pool except for a technical difference: if the submission is less than 2-3 days old, we re-up it; if it's older than that, we email a repost invite.
Thanks! I always assumed there was a bigger difference between "you mail me" and "you simple repost", but it makes sense that the time elapsed is a factor.
It was originally a factor for purely internal reasons—whether a story is kept in RAM or not—but I think it actually makes sense to keep.
I've never used the front page and usually do not find much of interest there. All the gold is found by monitoring /newest. Some might think that's a waste of time but for me I've found it to be extremely worth it, discovering projects, products and information long before it reaches the front as well as the stuff that never ends up there.
I often forget about /newest - particularly if I get distracted and spend too much time reading a link+comments for something currently on the front (=landing) page. I've been thinking about maybe building myself a personal single page view into HN (with the help of the HN API) which displays the front page and /newest page alongside each other. Getting the view to update itself every 5-10mins would be a bonus.
Specific examples, please!
I think your current efficiency is pretty good. I'm sure I miss some, but there's no longer much that I see in "new" that seems great but disappears forever. Your "second chance" pool (https://news.ycombinator.com/pool) seems to be working very well. Do you have a sense of what percentage of front page stories pass through it? A quick scan seemed to suggest that it might be as high as 20% (15-25 stories successfully recycled per day out of 100-120 front page posts per day). Seeing as it's basically invisible, that seems remarkably high.
It is remarkably high- maybe even more than 20%. I just ran a quick analysis and it came out more like 40% but I feel like "that can't be right"™ so I'd need to dig into it.

I still want to figure out how to build software for everyone to participate in this system, but have been stuck for years on "how would that not just be a clone of the upvoting system". That said, anyone who wants to nominate a story for a second chance is welcome to email us at hn@ycombinator.com. People do this, and we always take a look and usually put the submission in the pool.

The only thing that's kind of a bummer is that well over 90% of the time, people are just asking for their own material to be put in the second-chance pool. That's fine to ask, and we're fine to do it if the article passes muster, but it's not really in the spirit of the thing to use it for promotional purposes. I always feel better when people email about an article they have no personal stake in, but just found interesting.

Just because you've asked, but I don't really want to do so because the simplest answer is "well, that's just not a very interesting post/nobody really cared about it/you just missed it."

An example is: "Rust 2024 the Year of Everywhere?" — https://news.ycombinator.com/item?id=32945978. Decently well received, yet completely missed my radar, and I'm in total Rust immersion this week. That's just an example from today, I don't have other examples for you because I don't write them down.

Could it be possible that this one didn't last long on the front page because of #comments > #votes, and HN tends to downrank these as hot topics?

That thread spent 4 hours on the front page, so I don't know if not getting front page exposure was the issue there.

Lists of front page stories, ordered by the amount of time they spent on the front page, can be found by clicking 'past' in the top bar.

Yes, but "attention they deserve" is a poor phrasing of "I wish I'd noticed that, but failed to check HN when it was higher up on the front page, or failed to check the second page, or ...". (I assume here that you aren't missing stuff due to excessive caching of the front page, or hiccups in the "front page" algorithm.)

I'll guess that you're trying to stay on some sort of "I want to read all the interesting stuff...that makes the cut at HN" (and maybe a few other sites) diet. Vs. read everything interesting on the internet, which is hopeless. HN's size still makes that a pretty large intake.

> "Have we reached a ...point where...content relevant to your interests [emphasis mine] rarely reaches you?"

In theory, HN could try to categorize stories, so users could focus more on their own interests. How HN did that wouldn't need to resemble how "mainstream tech news" sites do it. Say, create a few categories that many stories fit into, and many folks often don't want to see. Then put some checkboxes in the user profile - "Hide these story categories on the front/second/etc. pages". Maybe add an "invert" feature, so when you're bored you can look at all the stuff normally hidden from you. With a limited number of categories, that might not imposed much server-side load. (Just cache all possible versions of the front page, and add "&categories=0x.." to the URL for each user?)

HN deliberately doesn't categorize stories; it's part of the premise of the site that there's just one pool of stories to draw from.
"past" is what you possibly want. At the top header.

And since a few/most comments don't understand this post check out 'past" and see if it represents what you thought HN was talking about yesterday

https://news.ycombinator.com/front

Ah to be honest i've forgotten there is more than the front page.

I've tried recently to click into the Ask sections more because I think that is the place that can have more value, collective discussion and the knowledge pool that is available here is incredible, this is what I find most disappointing about a lot of other resources, such as reddit, it often just becomes trash when the potential to discuss ideas and develop concepts / learn from others is the reason any of these tools excite me to begin with.

There are so many things I wish I had people to talk to with, to help try and figure them out, not necessary just tech issues, often places like stack overflow etc is a good place for that, but more, life, personal, career etc.

I'd love for the new posts to appear side by side with the home page posts. I keep forgetting about them, but every time I go there I find interesting stuff.

HOWEVER I don't think this means they need to (or even should) appear on the home page!

The elephant in the room is that you can't downvote submissions, only upvote. Which means there are posts that don't deserve to be on the front page that can't be removed from the front page, taking away spots from other posts.

In general, though, it seems like the first hour or two after the post is crucial, and if it doesn't receive enough upvotes in that window, it never will. So there's a lot of timing and "luck" involved.

You can hide a post you don’t want to see, but I’ve never verified if that opens another front page slot for you.
> The elephant in the room is that you can't downvote submissions, only upvote. Which means there are posts that don't deserve to be on the front page that can't be removed from the front page, taking away spots from other posts.

I don’t think this is entirely true. Posts can be flagged as an indication of downvoting (though there could be other reasons for flagging as well). When more people flag posts, they do go down in ranking. That’s my understanding and I do use flagging as a mechanism to push posts that I’d rather not have featured.

(comment deleted)
That's why I mainly access the posts through RSS https://hnrss.org

Personally I use 10 points as the minimum limit, that's usually a good indicator that a post already got some traction (yet it might be still removed or flagged later) https://hnrss.org/newest?points=10

I also use RSS on Inoreader. Helps me avoid scanning through the site for good content. But I think this doesn't help the problem brought up by OP
I did try hnrss, but there's like a hundred posts with 10+ upvotes every day. The sweet spot for RSS is rarely updated feeds, not a firehose.

Even 100+ vote threshold is still too much content, and at the same time you lose a lot of cool, niche articles.

> many times I happen upon posts in the 100+ votes from the past day or two that somehow I didn't even see

If the issue is posts that drop off the front page before they are seen, you might like using Wayne Larsen's https://hckrnews.com as an interface. It gives you a chronological list of all posts that get enough votes to break out of "new".

I use hckrnews as the main entry point for browsing HN, it I’ve noticed that hckrnews isn’t adequate because it relies again on the voting patterns of HN users. And most HN users would be voting for the ones already trending on the first and second pages. It’s the small (my guess) user base that browses through New almost all the time and upvotes or flags posts that’s doing the bulk of the work in surfacing things for others.
Personal anecdote: I've posted something in the past which, perhaps due to a timezone mismatch with the majority of users or something else, got no points and no comments.

A few months later I got an email from the mods inviting me to submit it again, since they saw it could interest folks in here. I did it and it got hundreds of comments.

So at least there's some mechanism to "salvage" potentially interesting stories, although it's a black-box to me :)

> perhaps due to a timezone mismatch

There's been a few blog posts about the best time to post. For example:

https://blog.rmotr.com/the-best-time-to-post-on-hacker-news-...

I'm aware of it, and I'm pretty sure many know about it, but I refuse to play this way.

In fact, sharing this information encourages others to game the system, lowering the overall quality of the content.

That was the Second-Chance Pool: https://news.ycombinator.com/item?id=26998308
Wow. This is great. How do you guys know these extra features of HN? Is there a site map of HN? So maybe one can explore the other pages that are not linked from anywhere on the homepage.
Not sure if there's a page listing all such features. I might have come across and bookmarked it somewhere, but can't find if there was one.

I remember pool because I visit it daily (after I found it mentioned in some thread).

At the bottom of the homepage, there is a link titled “lists” that links to many of these pages. Active is my favorite list as it points to posts with high discussion activity.
We explained some of the problems of the ranking algorithm in an article and backed it up with data: https://felx.me/2021/08/29/improving-the-hacker-news-ranking...

From that point on, we continued to research this problem and are working on an improved HN frontpage website right now, which takes live data from HN and factors out the feedback loops. It should be ready to use soon. If you want to get early access or get notified, please write me an email.

Great work, and well-presented findings. I'm really excited to see this in the wild.
Agreed. One aspect I’d like to understand better is how the dynamics will change. Eg today a highly popular post can stay up for a whole day. I get the sense that the new algo may flush things through more rapidly:

Consider two posts, with upvote rates p1 and p2 with p1 =2p2. That is the first is twice as good as the second.

Now if the second post is half as old as the first it would actually have about 2x the score of the first, despite being half as good.

This suggests more quality posts will get a chance to sit on the front page but each may stay less time — if so this would exacerbate the OP’s concern.

It would certainly be interesting to see an improved HN front page (assuming people could agree on what counted as improved). But how do you measure views? Your article talks about that and it seems to be part of your algorithm, but we don't publish that data, and in the case of article clicks, we don't even collect it.
I asked pretty much the same question at the time because it seemed like 'backed up with data' is a bit of a stretch. The answer then was 'it's actually not backed up by data'.

https://news.ycombinator.com/item?id=28405819

Yes, you are right. Only the existence of the problem is "backed up with data". The proposed solution is just an idea so far. I hope our implementation brings new insights.
Yes, the main feedback from HN back then was that we don't have any click-through or impression data. That is true, and we don't want to introduce any user tracking to make our idea possible. That was a very hard nut to crack, but after a year of research we finally think we have a practical solution: Instead of views, we estimate the attention a story has received. The main idea is to track how much time a story spent on different ranks. We created a causal statistical model to approximate the amount of attention a story gets on specific ranks. These are used to calculate a total accumulative attention number for every story. This attention value is then used instead of the views mentioned in the article. That's the main idea. A more detailed writeup will be published, together with the implementation of that frontpage.
Nope, not at all. Unless it's about my submissions. In that case, yes, 100%!

Joking aside, the only way to get to the front page is either via pure luck, and lots of it, or on a bring-your-own-upvoters basis. I think it's normal with the amount of traffic, users, and submitted posts that we see here. The front page became, over time, an expression of what the "general audience" of techies like - an intersection of their interests, not a sum. It's perfectly natural given that there are only 30 articles on the front page - the competition is fierce, and you can't get there by relying solely on people interested in particular content. It reminds me of elections: while you may be able to get something by relying on your die-hard electorate, to actually win you need to appeal to just about anyone and their cat...

> the only way to get to the front page is either via pure luck, and lots of it, or on a bring-your-own-upvoters basis

I don't agree. I hit the front page with ~50% of my own content submitted by me or others and I know others who also had good success tailoring their posts for HN users. Sure, luck plays its part as does the time of the day/week. But you can help the luck a lot by considering your audience.

Any advice on what to consider and how to tailor content for HN users? I've seen some advice before, but I just couldn't square these with the content I'm interested in, ie. following that advice would make the content into something I'm less interested in. Even if that worked, what would be the point?
From my experience - stories do better than explanations, things you can play with do better than text, slightly controversial things do better than plain, almost everyone is interested in a tech/business/novelty crossover. Some of this you can change without changing the topic itself. This is not even HN specific - just how to make writing interesting, which many authors don't.

Then again, I'm a sample of 1, so don't listen to me :-)

(comment deleted)
"I hit the front page with ~50% of my own content submitted by me or others"

How many submissions are you talking about? I only found this:

https://news.ycombinator.com/from?site=github.com/viraptor

Maybe your regular posting window coincides with the ideal posting window.

I rarely post, and when I do it's in early EU morning. 90% of the time, I don't get more than a single upvote, while I've found late US afternoon to have better luck. Not sure if European early birds are a tough crowd, or if Western evening/night time users are more generous.

> or on a bring-your-own-upvoters basis

That will probably get automatically and/or manually detected and get the submission flagged or killed and the user/site banned if they do it too much. From the FAQ: https://news.ycombinator.com/newsfaq.html

> Can I ask people to upvote my submission?

> No. Users should vote for a story because they personally find it intellectually interesting, not because someone has content to promote. We penalize or ban submissions, accounts, and sites that break this rule, so please don't.

> Can I ask people to comment on my submission?

> No, for the same reason. It's also not in your interest: HN readers are sensitive to this and will detect it, flag it, and use unkind words like 'spam'.

<popular person on twitter>: "Check out my latest post: <link to blog post on HN>."

They didn't ask for comments or votes.

The software may misclassify them as voting ring, or sockpuppets or shills or whaterver and still ban them. I'm not sure if there is an internal classification, because the details have never been published because they are part of the secret sauce.

Users may notice and then flag and/or email the mods.

Also, if the new users make a lot of silly comments like

> This is great! Is the best post/app I've ever seen!!!

it will increase the chance of making people angry and flagging the post.

(comment deleted)