Server is smoking, trying to figure out why (twitter.com)

My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things!

This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)

If the number of users has grown over the last 14 years, I'm sure some of the code areas have needed to be tuned over 14 years as well.

I think I've noticed changes a few times. For example,

- the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point.

- we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code.

- code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs.

- the "past" link in the navbar was added

- the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating.

These are relatively recent changes.

When you hit scale every edge case becomes a reality.

Ajax upvotes, upvote used to be just a link iirc.

Automated spam/voting ring detection and ranking systems.

That's what it says after you use the [-] to collapse a comment and it has a single reply. It also counts replies to replies (and so on).

I also saw that yesterday. What's worse, the only way to expand the comments hidden behind that seems to be to temporarily unblock Javascript, reload the page, and click on that.

Also assumed this, was only convinced otherwise when I ran into the same issue on my personal mobile phone, on a data network separate from my work network.

I wouldn't necessarily call it a bug, but the way the pagination works does not work well for active threads which is probably why they want to replace it at some point with code than can handle long threads on a single page.

A problem with pagination as implemented on HN is that the algorithm to display page N seems to be functionally (no pun intended!) equivalent to this:

  Render all the comments as one long virtual page
  Split it into smaller pages
  Render the Nth page from that set of smaller pages

Note! I'm not saying that is how it works. Just that it produces the same result as that.

So you read page 1 of an active discussion. You get to the end of the page and hit "more". But while you were reading page 1, other people were commenting, and other people were voting. Now, when it goes to render page 2 for you other comments have moved to the top, pushing the stuff you already read on page 1 down to page 2.

Or maybe what was just after page 1 when you were reading page 1 has moved up due to voting, and when you go to page 2 you end up past it.

It's like if you had a book that went through several editions, with significant changes between editions (including rearranging the order of chapters), and someone put together a printing by taking the first 10 pages from the first edition, the second 10 pages from the second edition, and so on.

This could be fixed while retaining pagination by saving a timestamp when you go to page 1, and then when you go to subsequent pages use a snapshot of the comment database from that timestamp instead of the current comments. If you want to update to the current comments you'd have to hit refresh or go back to page 1.

Another approach while retaining pagination would be keep track of what top level comments you've seen. If when rendering page N it sees that due to comment order changes one of the comments that is going to end on the page is one you've already seen it could omit it. Or maybe omit it unless the tree starting at that comment has changed since you last saw it. (If going this route, I'd also suggesting adding something to the UI to highlight what has changed in the tree and/or dehighlight what you've already seen).

That second approach could still be annoying. You could still get stuck on a single comment that is getting a lot of relies. That could be fixed, but then you are getting into probably needing to add new controls or settings.

I'm sure they have thought of both of those approaches and more, but probably want to keep the interface clean and simple and retain minimal state, hence the plan to someday make it handle very long comment threads on a single page.

A post from a week or two ago about APL and cellular automata (I cannot find it) perhaps illustrated OP's sentiment better: it's easy for a language to do something easily and elegantly, but quickly become fantastically difficult to accomplish with the introduction of new—and sometimes mundane—constraints. The post showed how it was trivial to implement certain cellular automata in APL, but massively difficult to do when the automata became slightly more complex (while being trivial to update a Java implementation to accommodate the new complexity).

The practical difference between running on one server with one core and five servers with four cores is nothing—to YC it's a rounding error either way. If there's a meaningful problem (e.g., comments being computationally expensive to paginate well) that the language/runtime/system/etc cannot solve easily enough for it to be remedied in O(months), it does call into question whether the choice of language is the right one. Yes, it can be patched over by a human moderator. But it still highlights a significant weakness in the technology's core function.

Perhaps ironically, most YC companies (and by consequence, YC) would fail miserably if they chose purism and principle over pragmatism when it came to technology decisions.