HN Explorer

Ask HN/PG: Why are comments being paginated?

341 points by roxstar ↗ HN
Is this a new experiment or is it a new permanent thing to reduce load, I don't think many people are going to be looking at the lower end of the spectrum (clicking more) which may detract from interesting comments and conversations.

Examples: look on the front page with comment threads >~30

121 comments

[ 4.4 ms ] story [ 186 ms ] thread
msbarnett
I agree, I think this basically disincentivizes commenting when you know you're going to end up "below the fold".

I went into the feature request thread to see if anyone had requested the ability to turn this off, but gave up on looking after rapid-fire clicking "More" 25+ times. Content more than a page or two back might as well not exist.

jedsmith
The Reddit community actually speaks of this phenomenon in threads addressed to new members ("things you should know about Reddit", etc). They say that once there's several long threads with a lot of upvotes, you're better off not commenting as your comment will most likely languish without any attention whatsoever.

I'd say that's partially due to scrolling, but you're correct - that More button is so non-obvious that I didn't even see it until I read this thread. Those comments below More might as well not even exist.

The most common retort to that viewpoint that I saw was "well, so you won't get karma. Bummer. Your opinion is out there anyway," which is missing the point, I think. It's not about the karma, it's about contributing to the conversation. Now, contributing to the conversation is a competition based on time and popularity.

This change will probably do two things: hide a lot of good comments below page flips, and cause people to quickly comment on stories to fill above the fold.

drivebyacct2
Funny, I find that I can reply to a comment, even when it's childrens' comments number in the hundreds, and it still gets attention.
dauphin
Naab it's because of shitty posts like yours than HN is slow! New users should not be able to post. Abusing the system is a problem.
jedsmith
[Edit: The comment I replied to here originally had very different content about threading and scoring. I don't know what it is now, so I've removed the reply.]
zbanks
I don't know how reddit works exactly, but I notice that they generally have a few "hidden" comments in the tree.

I assume this is to prevent this phenomenon. If your comment is significantly worse than the rest of an active thread, it shouldn't be prominently displayed.

jedsmith
The 'hidden' comments are based on a per-user threshold. My brief Reddit experience improved significantly when I raised said threshold.
repsilat
I agree that there's a lot of fluff on Reddit, but I don't think I agree with your solution. I find a lot of good, technical comments end up quite a way down the page in more detailed discussions between users who are more interested in the subject matter than having a conversation (or having their joke voted up the page.)

Really, I wish Reddit had something closer to Slashdot's moderation categorisation to filter on. Filtering out "funny" comments would be nice. Ignoring votes cast by people who just want to register their agreement or disagreement would be an incredible, impossible goal.

drivebyacct2
Well, jed has removed it, but his original comment here implied that links|comments are ordered solely on the score via voting... While that is a larger driver in the "topics" that get posted, it certainly is NOT the algorithm used for determining comment ordering.

I noted more in my later comment responding to your (in my opinion) naive cheapshot on reddit, but the "best" algorithm, that is the default, does not simply rank by the number of votes. And even then, the nature of subreddits and people opting to subscribe to that which they're interested in goes a long way to putting their "upvotes" or "downvotes" into the context of the users in that subreddit.

jedsmith
Your summary of my comment is nowhere near what I wrote. I'm paraphrasing from memory, but I believe my comment was something like this:

> That's a problem with any site that threads comments and sorts by score.

The parent to my comment pointed out a corner case with scoring that he felt was specific to Reddit, if I recall, and I replied to indicate that such a problem is inherent to sorting/scoring at all and not Reddit itself. Since I made that reply (which actually gives Reddit credit as saying said assertion isn't specific to them), the parent edited to say something else, and I'm not sure how to grok it.

Rather than delete and look extra shady, I decided to leave that there. I'm glad I did, because your edit here in response to that speaks volumes as to why you chose to respond as you have.

First, I'm well aware of how sorting works on Reddit. I've deployed my own copy of Reddit's code and successfully brought it up, and I've manipulated the code and looked around in it (I'm a Python developer by trade).

Second, one of the problems inherent to text is that meaning can be lost, very easily. I wrote the comment to mean "that's inherent to the style of commenting Reddit chose, not Reddit per se". That's a comment regarding a concept far abstract from specifics of Reddit's voting constraints - it's a comment addressing the concepts of threading and scoring themselves.

You took my statement as "Reddit has that problem, and it sucks because". I apologize that I wasn't completely crystal clear about what I meant.

Even after rereading what I wrote, I cannot see how the way you've taken it could be considered what I intended -- you're really reading a meaning that you want to read, because you want to pick a fight over someone disparaging Reddit. I was a regular contributor to Reddit for several months and realize its value. I think it's a great community. You're not being a great representative of it at the moment, but that's your choice, not theirs.

Since you edited out 'slander', I'll put it back:

> naive cheapshot slander on Reddit

Even with the aforementioned misunderstanding, which is somewhat forgiveable, jumping to slander is a bit much. You know this because you edited it out. I'd ask the courtesy in the future of assuming the best in people, and not accusing them of torts when there's any ambiguity to meaning that you might have missed.

drivebyacct2
Well, as you have removed it, I can only speak to my interpretation, which as you imply, may have been off-base, and if so, I do apologize.

Please don't take me as some white knight of reddit, I think that we may agree more than I originally understood. I took your comment as a dismissal of reddit purely on the basis of voting. I apologize as it seems that may not have been your intention. I merely sought to explain the other mechanisms that reddit uses, in addition to the automatic (imo) value of subreddits, to determine the value of content.

>I'd ask the courtesy in the future of assuming the best in people, and not accusing them of torts when there's any ambiguity to meaning that you might have missed.

You're of course correct. As you noted, I editted my comment as I tried to take a more understanding ground. I'm not used to people reading so quickly. I have a tendency to rephrase and uh, un-embelish, after a second or third evaluation of the thread. I do apologize.

Though, I suppose in the end, I still simply don't understand the alternative to voting on content and I feel like the directed subreddits are add relevancy to those votes. The alternative is all moderated or selected content? A community where comments can be killed (ahem)? I guess I like the anarchist communal democracy, even if it ends up that a bunch of Internet kiddies want revenge for a tortured cat. In my mind, at the very least ideologically, it's always superior to the alternative.

drivebyacct2
I'd love another shot at this conversation. If you grant that there are other factors at play for determining comment ordering, and you still insist that a site is "broken" because it utilizes voting... what is the alternative? Or are you suggesting that all sites are in some fashion "broken" and that it's a common problem with any site powered by user content?
Joakal
Reddit solves this by hiding poorly ranked comments.
jedsmith
Not quite. That filters out the bad ones, but the underlying problem still remains: people reply to highly-ranked comments with something only tangentially related, or not related at all, in order to put it in front of a wider audience.

People then upvote it even though it isn't related because it might be insightful.

[dead] ezalor
Sullivan's paradox.
[deleted]
(comment deleted)
drivebyacct2
Please don't just kill this again -- I cleaned this post up as my wording the first go around was unnecessary, but I think there is a valuable point here.

I don't find this to be the case at all... though I do find it to be the assumption made by many who don't actually use reddit. I just checked the top few threads to ensure I don't have confirmation bias (heh, though I understand the futility of such an exercise). Reddit is VERY good about not letting people piggy back on top posts and their "best" algorithm does not sort according to score. It sorts according to how those who voted generally vote, and based on how the majority has voted, both up and down, regardless of overall positive or negative score for the post.

The notion that replies are contextual relevant to their parent comment on reddit is a TESTAMENT to my original claim:: That reddit does a good job of exposing, even new posts, in inundated threads.

(As usual, it surprises me how much if the "our community is better that there" rhetoric goes on here, while simultaneously, we seem to mock that idea of communities making that judgemental statement.)

edit: And even so, is it a bad thing if non-relevant intriguing content is upvoted? Just click the [-] if you don't want to see it... but that's the very nature of discussion. Things go off topic and humanity learns as a result.

rflrob
> People then upvote it even though it isn't related because it might be insightful.

What if we, as a community, decided that deliberately trying to float with highly rated comments by posting completely unrelated "responses" is not something we'll tolerate, and downvoted violators, regardless of the insightfulness of the comment? Tangential responses are something more of a gray area, but I'm willing to let them be voted upon on their own merits.

drivebyacct2
I meant on reddit. Is your reply satire or am I missing something?

I'm not a new user and my comments generally fair well here.

I (was, but am not now) downvoted on HN for saying that my posts on reddit get well received attention even with lots of noise... Seems ironic...

nostrademons
On the plus side - this encourages people to check out the New page and upvote stories they find interesting there, so that they can get in on the comment threads before they've become hot.
silentbicycle
It could be counterbalanced by some sort of notification (on the Threads page, especially) for replies. I don't care if it's fallen off the front page, if somebody is carrying an interesting discussion in a thread in which I've commented, I may still like to know. Even if there are several days between each comment.

The site's incentives to only comment on the hot threads are probably detrimental.

JeremyBanks
You can get reply notifications (email or mobile) using Notifo (YC W10)[1]. I think it's only for replies to your comments, not for all activity in a thread.

  [1]: http://news.ycombinator.com/item?id=1720763
riledhel
Maybe we need something like twitter implemented... when you scroll below certain number of comments they load n comments more automagically.
regularfry
That kills the scrollbar. A better way to do it would be to show far fewer comments per page, so that you get approximately a screenful at a time, then make the "next" and "previous" links huge and blindingly obvious. I think you should scroll or navigate, not both.
flatline
The more link has the rel="nofollow" attribute, so anything but the top comments will (in theory) not be indexed by search engines either. Given that some functions of the site are based on karma, I hope it's just a stopgap measure.
nostrademons
I believe rel=nofollow only means that the link carries no weight for PageRank and other signals that may affect the target URLs ranking, not that it won't be indexed. To block indexing, you need a <meta name=robots content=noindex> tag on the target page itself.
bendtheblock
If performance is an issue, some sort of lazy load would be better, so just before you reach the bottom of the page more comments are loaded.
thecoffman
I feel like it would be more tolerable if it loaded the new comments inline a la reddit: at least you wouldn't lose the context/continuity of the conversation that way. Unfortunately it appears that its actually spreading the comments across multiple pages so that when you click "more" you lose everything you were just looking at.
[deleted]
(comment deleted)
Flenser
I think this basically disincentivizes commenting when you know you're going to end up "below the fold".

And incentivises replying to comments above the fold.

blahedo
Perhaps more importantly, it seriously skews the karma that accrues to responses---those that are above the fold will continue to get upvotes while those below the fold get none, so the disparity grows.

I've seen this effect on other sites; one gaming site[0] shows two views of the comment list, "recent" and "top-voted" (both with a "more" button that few people seem to click)---the "recent" comments rarely get more than two or three votes before scrolling off, while the difference between the lowest upvote count in the "top-voted" count and the next-highest comment can be in the hundreds or thousands once a game has been up for more than a day or two.

[0]Kongregate.com, if anyone's curious.

zppx
Let's wait for the official answer, but I also include my take.

Maybe because of the community growth. When I began to hang out around here, 6 months before I created an account I believe, 40 upvotes was a huge amount for a post or a comment.

Today it's common to see posts with more than 100 in the front page and comments receiving 60 or so.

EDIT: The number of comments in each posts also exploded, 20 comments in a thread used to make it very active.

EDIT 2: For clarification.

gscott
Your more likely to upvote to get comments that you like higher. I typically do not upvote much but now it seems important.
kmfrk
I didn't even notice that - that's incredibly annoying.

I'd be fine with it, if the More returned the rest of the comments, not the x next comments.

I don't really see the point of this either; it can hardly be that big of a resource hog on either ends.

staunch
I'd rather the page take 10 minutes to load than have it paginated. I know PG is just trying to implement a quick temporary fix, but this one is so annoying it made me go back to work (gasp!).
eam
Yeah, horrible feature. I get lost where I'm at. I'd love to see it back the way it was please. :)
alttab
It would be interesting to see an auto-extend feature like Facebook does with the timeline. Keep scrolling? We will AJAX load them into the page in batches. That should do ya. No buttons necessary, full comment history the further you go, and you don't show more information than you originally ask for (just a couple of comments).
sans-serif
As long as it doesn't grow at the same rate as Twitter's, this should scale.
cabalamat
I very much hope pg doesn't do this; I find it very annoying.
danvet
As some other commenter already noted, this breaks browser text-search. In case you want to quickly checked whether what you want to say has been said already (and upvoted that comment instead or add something as a reply).
shawndumas
Until it gets reverted (hopefully) -- AutoPagerize [1]

----

[1]: https://chrome.google.com/extensions/detail/igiofjhpmpihnifd...

jamesbritt
I'm curious: Why does this plug-in need access to my browsing history?
Joakal
That's a bit bizarre to comment when the primary company that developed Chrome is known for keeping tabs on users' browsing history.
ezalor
wat do you mean?
Joakal
If you type into Chrome's URL/browser bar, it sends the query to google's servers which in turn returns a list of possible search suggestions (autocomplete style). Google likely retains the query for commercialisation purposes.

So the user being worried about a Chrome plugin sending browsing data seems bizarre.

jamesbritt
So, when I click a link on HN, using Chrome, Chrome is sending that information to Google? Likewise when I type http://somethingorother, that's going through Google first?
johnswamps
It's probably requesting "tabs" permissions. The Chrome security permissions aren't fine grained enough so often you need to request permission for a lot more than you actually need. For some reason requesting "tabs" shows a scary warning about having access to all your history. You'll find that a ton of extensions need access to your history for this reason, it's actually really annoying as an extension developer.
Encosia
That's the priveledge that Chrome extensions require in order to inspect URLs as you load them, so they can conditionally apply themselves to the appropriate site(s). As far as I know, there isn't a more granular priveledge to allow access only to URLs as they're loaded (which an extension could still use to track your browsing habits by storing them in its localStorage).
requinot59
Using Ajax for "load on scroll" may be a solution. Or go the reddit way, don't paginate but don't load full sub-threads.
lwhi
I noticed the same, and assumed that it was a measure designed to improve site performance.

Recently, I've found the HN site has become pretty unresponsive at times - I imagine limiting the number of comments on each page is going to reduce the burden on the server.

forkandwait
I like plain text emails, terminals, emacs, and I really prefer un-paginated html pages.
ratsbane
Often I read HN with Lynx. The plain HTML here works much better than the markup at most other sites, including Reddit.
redthrowaway
There's a really easy fix to this: Load more comments on scroll-to-bottom. No button, no bandwidth used, but those who make it to the bottom of a thread don't have to do anything to see new comments.

I agree with OP; the current system is suboptimal, to say the least.

cincinnatus
That would be the first instance of an ajaxy thing on HN I believe.
[deleted]
(comment deleted)
ianl
The upvote/downvote on comments and stories uses ajax i believe.
requinot59
yes/no: it uses simple assignations to (new Image()).src.
nfriedly
Not sure why you got hit with the down votes, I always assumed it used "real" ajax.
antimatter15
Infinite scrolling screws with scrollbar behavior though :(
redthrowaway
Not until you hit the bottom of the first "page", at which point you've resigned yourself to more content.
flatline
Ugh, I hope this does not happen, I hate the scroll-to-the-bottom-and-get-more thing. Particularly if the server is under heavy load already, you never know if there is more or if so how much, and it takes the scrollbar away from your mouse pointer once it's loaded another chunk due to the relative offset.
brianpan
I'm confused by your reasons. With scroll for more, you would know if there is more- the end is the end, just like with pagination. And it's no worse than pagination in the "you don't know how much more" department, either. The pagination is a "More" link, it doesn't indicate how much is on the next page.
regularfry
One would think it would be relatively trivial for "More" to become "256 More" or similar...
mnutt
Maybe I'm missing something because I've never seen it before, but has anyone ever tried leaving blank space at the bottom of the first page relative to the total number of comments? It may work better on items of fixed length than comments, but even on comments it would at least make the scrolling a little less jumpy.
marshray
NOoooo!!! Anything but the Scrollbar of Sisyphus!
1337p337
I'd not heard this term for it before, and it sums up the horror accurately. You have enriched my life.

The Scrollbar of Sisyphus breaks the back button (click "Back", browser remembers where you had scrolled to, can't go that far down, lands at the bottom, triggers second "page" of results) jumping to the bottom of the page, occasionally searching, and forces you to jump to the end of the page repeatedly to figure out where you were. It breaks nearly every UI expectation one has about scrolling (especially on a phone) and several that your browser has about caching.

redthrowaway
Those problems you listed are implementation-specific, and not necessary conditions of the feature.
mambodog
This still would prevent being able to CTRL/CMD-F for a particular term to see if anyone has mentioned it already, something I do a lot, and I imagine others do too.
redthrowaway
This is true, but a little bit of js would take care of that.
lotusleaf1987
Is the feature really saving that significant of load time? I mean HN is as minimal as it gets, I'd rather all the comments load and take 3-4 seconds more than have to manually go through and hit more several times.
1337p337
As pointed out by pg above, it seems not to. An educated guess (read: wild speculation on my part) indicates that, due to the ranking algorithm of HN (versus Twitter's simple sort, where new things show up first and order never changes) rendering and bandwith time are probably dwarfed by the time required to load and sort all of the comments on all pages to determine what comments show up on the first page.
klbarry
It might be a good idea for PG to charge a small monthly fee if the issue is server strength.
[dead] dauphin
or use a new genius p2p system to serve HN! Telehash, here i am!
jedsmith
I'd rather they look over the architecture and possible changes to the codebase instead of resorting to charging the users.
kentosi
There should at least be a feature within our profile settings to disable this.
oomkiller
Of all the features we need on HN, this is not one of them.
nhebb
It's harder to see whether someone has already made the same point that you want to make, so it could lead to redundant commentary. I often do a quick keyword search of the page to see if someone has already made a related comment. With pagination, you can't easily do that.
solipsist
Everyone should be using AutoPagerize[1] (a Sarari extension) by now, or at least something similar. It makes your life a lot easier on most websites, one of which is now HN.

[1] - http://autopagerize.net/

dimarco
Couldn't people take advantage of this by only commenting on high-karma comments, regardless of whether or not it's relevant to the parent-comment, but instead just piggy-backing to stay on the first page?
wvenable
There is no meaningful information in this comment because nobody is going to see it anyway.
zaphoyd
When did this start? I read almost exclusively through the ihackernews.com mobile site. I just checked and it does not implement the more button and only shows the first 40 or so comments. It looks like I have been missing out on the end comments of popular threads without realizing it. :(
[deleted]
(comment deleted)
cousin_it
I don't understand the rationale for this feature.

Here's a better way to reduce load: make commenting not require a reload. Same for editing comments, deleting them, etc.

invisible
I'd rather see a hard limit on the number of top-level threads allowed than a "More."
pg
To cut load. I'm not sure it helped much though, so I may not keep it.
tommi
Good. Your comment is the most valuable answer in this thread and it was the last one to show on first page.
estel
Indeed, I had to go below the fold.
seancron
I almost missed this comment because it was just barely above the fold.
cstuder
It's actually below the fold now.
acqq
How about displaying the comments "shallower": if some discussion ("a branch in the comments") goes too deep, maybe that's what doesn't have to be on the same page, as it's obviously something where a lot of arguments and counterarguments occur, and not something where the biggest insight is to be expected.

Such comment handling would however certainly motivate people to "post without the parent."

danvet
I'd very much welcome that. My gut feeling is that on quite a few posts, discussion seems to stick to the first comment. I'm assuming that starts to happen as soon as this top-level comment has enough upvotes to quickly displace any new postings.

I think preventing such "sticky" threads from clobbering the first pages of comments may add some decent value to the discussion. Especially in brining in different aspects and povs, perhaps sometimes yielding a more balanced discussion.

nitrogen
Is there some way the HN community could help with the load, or some pathological behavior that a lot of us are doing that could be replaced by behavior that's easier on the server(s)?
Eliezer
Er, that doesn't seem like a good reason to downgrade the UI of something important. I don't know how much money it takes to keep HN running, but would just buying ten times as much computing power really be a significant expense for YCombinator, compared to the dealflow from HN?
akkartik
I suspect the bottleneck isn't cost so much as the mental bandwidth to switch hosts, etc. It would be a context switch for PG/RTM/bringing someone else up to speed on arc. HN's just naturally lagging other priorities.

---

Farther in the future: HN's arc webserver uses simple flat files for storage, so moving from one host to two is probably a big context switch as well. I wonder what the current server's specs are, and how long we have until that kinda-sorta-Y2K-like barrier.

axod
or rewrite it in a more efficient language/runtime would likely have more effect.
pg
As far as I know, the current server has close to the fastest processor available.
akkartik
RAM might matter more for news.
gokhan
If you decide to keep it, please keep displaying the header on subsequent pages. Ajax loading the rest on the first page is better, though.
paolomaffei
it's fun that this answer is below the fold. I wouldn't keep it.