HN Explorer

Please fix: Unknown or expired link.

112 points by kijin ↗ HN
I'm pretty sure that the HN community likes to read thoughtful, thoroughly researched comments that are somewhat longer than one-liners. But whenever I take more than a few minutes to write a comment, I am greeted with the following error message as soon as I hit "Submit":

    Unknown or expired link.
Fortunately, I can usually hit the Back button, refresh the page, and resubmit my comment without losing the contents of the textarea. Anyway, I try to be extra careful and copy the comment to the clipboard before I hit "Submit" if I feel like it's been more than a few minutes since I started writing.

But this should not be necessary in the first place.

Please increase the timeout on your CSRF tokens (or whatever it is that triggers the error message above) to something more user-friendly. I would suggest a sensible default of 86400 seconds.

Thanks.

76 comments

[ 5.5 ms ] story [ 182 ms ] thread
lutusp
There's a reason things are set up as they are:

* Although HTTP connections are normally stateless, HN has a login system for account holders, which means there's a state, a connection, associated with each logged-in user.

* Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.

* To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use.

* There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use. Also, heartbeats would represent an extra traffic load on the server. And now that tabs have become popular, there's no way to be sure that a heartbeat response isn't from an unused, forgotten tab among dozens.

* Therefore, after a connection has not seen packet requests for a set time interval, the server expires the connection to make it available to other visitors.

* The shorter the timeout, the more connections the server can handle.

> I would suggest a sensible default of 86400 seconds.

Let's say the server can handle 2048 simultaneous login sessions (I don't know the actual number). According to your suggestion, after the first 2048 visitors log in, the server would refuse to accept any new logins for 24 hours. This isn't realistic.

Here's how I handle the problem -- if I decide a reply is going to take more than a few seconds, I transfer my reply to a text editor and let the connection expire. When my long-winded reply is ready, I refresh the original page, paste my reply in, and click "Add Comment".

I hope this helps.

krapp
Although - other sites can manage user state and validating posts just fine without this 'feature.'

Even assuming it's all but impossible to fix given the implementation in Arc, surely it would be possible to automatically reassociate the post with whatever the current function id is? It seems to be as much a user experience problem as anything.

jayrox
i always figured it was because they built each page and cached it to html, then that cache was set to expire after 5 minutes. then once the cache expired you couldn't go to the next page or what ever because the cached file was deleted.

IMO the reasons presented for this "Feature" should be changed to "lazy programming". cause that's all it is.

krapp
Even then it would make more sense if the location that new comments were POSTed to was not necessarily one which could expire. Would it be impossible for the API (or whatever passes for an API) to present a static endpoint for receiving comments?
jayrox
its not impossible, just appears to be coded by someone who over did it with caching in the beginning before the site needed it.
exelib
Then it's broken by design. There is no sense to do so, but a great way to anger visitors and destroy UX.
nwh
I don't think this is what is going on here, it makes no sense for it to operate like this. To demonstrate this is not the case, I just momentarily unplugged my NIC and then submitted this comment. The connection was definitely dropped and re-associated, but my comment passed perfectly fine. This is more likely a very short cache time on CSRF tokens rather than anything to do with associated sockets.
phamilton
Http is stateless, so there are no "known sockets". Every request opens a new socket. Unplugging and plugging back in is a no-op in this context.
brudgers
The architectural model your comment and the initial post assumes to underlay the behavior is incorrect.

The reason for 'Unknown or Expired Link' is that HN runs on a webserver that uses continuations to maintain state. The continuations are stored in a table and based on load individual table entries get flushed to make room for new entries as needed. The message is actually always the result of an unknown link because the hash is no longer in the table.

Your experiment worked because the relevant continuation [state reference] was available after disconnect e.g. the 'route' so to speak was still valid.

Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. Until rather recently, HN ran on a single server. It still uses relatively few lines of code.

acdha
> Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language].

This is really the key point: continuations are the wrong way to write this kind of application. It's significantly more work and there's no benefit from doing so but it allowed someone to test a pet idea and, having done so, fixing it would mean admitting that doing something because it's cool isn't good engineering.

brudgers
The kind of application is specifically Hacker News, not a generic website or webserver. See the essay linked here for what kind of application this is:

https://news.ycombinator.com/item?id=7651525

acdha
Continuations are a questionable fit for web applications in general, however. HTTP is stateless and it requires a lot of extra work to build an abstraction which pretends otherwise. I prefer to embrace that from the beginning so the entire design matches the nature of the medium.
brudgers
The explanation is mine. It wasn't an argument.

The application and implementation decisions are YC's. The essay explains their rationales.

nwh
I know, I was demonstrating why that given explanation couldn't be true.
Quai
This is not the case. Websites, including hacker news, does not handle sessions like that.
kijin
CSRF tokens != login sessions != persistent HTTP connections.

There is no need to keep an HTTP connection open in order for the server to log people in or recognize when a valid token has been submitted with a form.

grey-area
Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.

You're making excuses for a particular setup - this is not something which holds true of all web servers. It's quite possible to store the state necessary to allow 10000 simultaneous logins if you use cookies and a db to store your state, rather than relying on in-memory state and fragile continuation links like this (created a few minutes ago):

https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw

or tying replies to stored state on the server which expires within minutes. Many large scale sites manage it, I suppose because they don't store data like this in memory or urls. I'm not sure about this idea of storing state in urls, it often backfires, and urls should identify resources and actions, not store state.

There is no good reason the server cannot receive a POST request from an authenticated user against a node id, and perform the requested action (post my reply please), no matter how many intervening logins/actions it has handled.

Looking at the form in question (reply forms) reveals why it does break:

    <input type="hidden" name="fnid" value="UswPVmT3a8lZnqPBtcx359">
The server stores a continuation in the comment reply form, which (like the more link above) expires very quickly. The reason it expires is presumably because the server keeps a list of these fnids in memory, and as you say the list is finite, but it doesn't have to do that. There are other ways to create a CSRF (if that's what they use this for) which won't expire so quickly, and this problem is not one which many sites experience.

It is annoying just how quickly reply links, more links etc expire on this site, and if the site used a different method to sign reply forms or construct other links they could get rid of this issue. Keeping data in memory like this will become more and more of a problem as the site scales - eventually things would break in seconds with enough load - that's clearly not acceptable.

zackbrown
A login system does introduce a notion of state between the server and client, but the use of the word 'connection' to describe that is probably a misnomer. I'm unaware of any basic HTTP servers that use a persistent socket connection (which seems to be the sort of connection you're describing, given the resource constraints you suggest) with clients in order to manage authentication.

Most often, the server just needs to set a flag associating an auth token with a given user (for whom it has a persistent representation, e.g. in a database.) Lookups against this token => user hash/map/table should be very cheap, especially if it's kept in memory. Further, even for tens of thousands of concurrent users, that's not a lot of memory overhead (1KB/authed user would be a generous upper bound, and that would still only be 10MB of RAM per 10k concurrent users.) Could be that HN is running on a 1990s-era desktop with 256MB of memory in someone's office and that this is an actual resource constraint issue--but my hunch is that this problem lies on the software side, not the hardware/resource side.

newobj
I'm not sure what's more disturbing; this reply, or the fact that only 2 out of 5 replies to this reply attempt to refute it.
ordinary
This is largely correct. Each reply form includes a hidden input field with the name 'fnid' in their form. Many other pages include a parameter with the same name. This is the unique identifier of a closure that's associated with the page you're on. These closures are deleted after X time, at which point requests from that page can no longer be handled correctly.

Yes, obviously it would be great if X were increased. But doing so would require an increase in server resources or a significant change in the way HN's backend works; neither of these options is free. That is the reason why things are this way.

This is an old debate. See, for example: https://github.com/HackerNews/HN/issues/11

grey-area
fnids are evil, they just need to drop them and respond properly to posts without them. What more state needs to be recorded which is not already encoded in a POST reply request {user:n,post:7651593,text:'xxx'}?

The solution should be pretty straightforward:

    Reply forms -> Remove fnid, authenticate on post, add CSRF

    Flag links -> Remove fnid, authenticate on post, add CSRF
They already check authentication on these actions anyway as you'll see if you try upvote links from another user without login, all that would be required would be to use CSRF instead of these closures to handle posts like this - the closures are not a workable solution on a large site, and I'd contend are dangerous on a small site - they throw away a lot of the advantages of HTTP for no appreciable gains.

Things like the more link are particularly pointless, they could just use something like:

    https://news.ycombinator.com/x?fnid=xxx -> https://news.ycombinator.com/news/2
This would change over time like the existing home page, but there's no reason for a simple GET request like this to expire.
quasque
The fnid is a CSRF token, just with an disadvantageous implementation that leads to a far too short expiry time.

I don't know much about the internals of Arc, but maybe it would be possible to serialise the continuation data, combine it with a reasonable expiry time, apply some form of authenticated encryption to this, and supply that as the fnid? And reverse the process - with appropriate integrity checks - when the fnid is submitted. Then you have your state distributed at the client-side instead of all being kept on the server, so it can scale more effectively.

EDIT: Nevermind, just realised that this very suggestion is addressed here, and it's more difficult than I anticipated https://github.com/HackerNews/HN/issues/11#issuecomment-3215...

grey-area
A CSRF token doesn't have to be stored in memory on the server like these fnids, you only need a way to verify it like a secret for decoding it. This is a solved problem for many other servers so they could just look at some other implementation of CSRF, or start by getting rid of the many uses of fnid which are not required anyway (they are also used on GET requests).

I don't think they should try to serialise fnids, but just get rid of them completely. What state is required for a posted reply other than the three things outlined above + CSRF protection?

quasque
You're right, I was overthinking the solution.

I think the most common technique I've seen for generating CSRF token is to compute an HMAC of the immutable request parameters.

I'm guessing that's what HN already implements for voting, as the token is dependent on user id and the id of the thing being voted on, and kind of looks like an SHA-1 hash.

acdha
> * Because of the state information associated with logins, A Web server has a limited number of login connections it can handle.

a) this is technically incorrect – you can trivially handle this using signed cookies to store the logged in user

b) even using traditional database or filesystem based approaches, those limits would be aggressive even if the site was hosted on an iPhone – you're talking tens of bytes per session.

> To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use.

Also technically wrong: if you aren't doing anything, you aren't causing any load whatsoever on the server. This would be different for something like a chat site using WebSockets but for a vanilla app like HN this is not an issue.

Also, again, even if you were using a persistent connection the absolute technical limits are much, much higher than the traffic HN sees - hundreds of thousands of simultaneous users.

> There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use.

Again, technically wrong: plain HTTP sites don't maintain persistent connections but in the case of something like a WebSocket connection you'd simply use TCP keepalives, which are a mature standard feature and very low overhead.

You appear to have confused the concept of a session with something which is actually happening at the network level. Try reading something about the HTTP request cycle (e.g. http://www.slideshare.net/cczona/full-stack-full-circle-what...) for basic background and after that move on to some discussion about the work done to make servers which can handle many simultaneous connections - the C10K problem was the original work tuning operating systems to handle tens of thousands of simultaneous connections and was large a non-issue at the network / basic HTTP leel by the mid-2000s:

http://bulk.fefe.de/scalability/

By now a single server can be assumed to handle hundreds of thousands of simultaneous connections at the network level – assuming, of course, that you have capacity to actual do whatever work those connections are used for. Idle users typing text into a form wouldn't be a problem even if it did use a persistent connection.

jamesmoroni
This happens with the next page button as well if I've been on the page for more than 3 minutes or so.
Quai
I see this almost every day. I have always assumed that Hacker News is too cool to fix/support navigation after the frontpage (or any other page) has changed.
judk
It's not "cool", it's that the implementation is a quick hack, not intended to be a quality product. It just needs to be good enough to make users grudgingly return and see the PR for YC startups.
midnitewarrior
Ya, for a community that is supposed to value great design and technical acumen, this site looks like somebody's pet project that was abandoned.
danford
I don't really mind the design. I have a feeling they purposely keep the site design simple and, what some might consider "unappealing" to the average user. Helps weed out the normies.
judk
It also weeds out the adults who think about what they read and don't have child eyes.
nitrogen
You can always zoom in or enable high DPI mode if your browser supports it
6cxs2hd6
Or one can always make a mother*ucking web site [^1], which works perfectly in every browser. Zero issues with font size, layout and scrolling. Just-works.

[1]: http://motherfuckingwebsite.com/

vannevar
This 'bug' has been around so long that at this point I suspect that it's really just a long-running troll.
chrismorgan
This has bearing on my absolutely favourite part of the Web Content Accessibility Guidelines (WCAG) 2.0:

Section 2.2.1, Timing Adjustable (http://www.w3.org/TR/WCAG20/#time-limits-required-behaviors):

----

2.2.1 Timing Adjustable: For each time limit that is set by the content, at least one of the following is true: (Level A)

- Turn off: The user is allowed to turn off the time limit before encountering it; or

- Adjust: The user is allowed to adjust the time limit before encountering it over a wide range that is at least ten times the length of the default setting; or

- Extend: The user is warned before time expires and given at least 20 seconds to extend the time limit with a simple action (for example, "press the space bar"), and the user is allowed to extend the time limit at least ten times; or

- Real-time Exception: The time limit is a required part of a real-time event (for example, an auction), and no alternative to the time limit is possible; or

- Essential Exception: The time limit is essential and extending it would invalidate the activity; or

- 20 Hour Exception: The time limit is longer than 20 hours.

Note: This success criterion helps ensure that users can complete tasks without unexpected changes in content or context that are a result of a time limit. This success criterion should be considered in conjunction with Success Criterion 3.2.1, which puts limits on changes of content or context as a result of user action.

----

Yea, verily! HN isn’t even Level A (the most basic level that "everyone" should comply with) compliant. Through this, HN fails Accessibility 101. (Its text colour choices are also often dubious as far as contrast goes—that’s covered in WCAG 2.0 as well, of course.)

I simply don’t care about the technical reasons. The choice of coroutines to back all of these things has broken the basic accessibility of all the content.

zura
> The choice of coroutines to back all of these things has broken the basic accessibility of all the content.

Exactly. This is actually a good example of More is Less.

seabee
The choice of coroutines is not the cause of the problem, it's the decision to store session state on the server and clearing that state before the user is done.

This same problem can happen in a PHP application, and it can be avoided in a Racket web framework application that uses continuations.

IgorPartola
Also, can someone please make the site render with a larger font on mobile? I am spending a fortune on new glasses and contact lenses, as well as monocles and loupes just to read HN when not on my computer. So many people on here have asked for this over the years. Perhaps this is a subtle attempt by the YC to make sure all the people in our industry have to use eyewear so we can recognize each other at the secret meetings. Maybe this has something to do with Gunnar Optiks marketing eyewear for geeks? Maybe YC is about to fund an optometrist practice. Or maybe someone here knows that Apple is about to release a 20 inch iPad?

Please, for the love of everything that is good and pure and true, make the font larger!

Sincerely, Blind among blind

Edit: please don't suggest one of the other HN wrappers. I know they exist. I don't want to hand over my HN password to them. My HN identity is important to me, so unless I can run my own wrapper on my own servers I don't want it.

Edit edit: I believe if HN mods/operators announced a contest for best mobile design they'd get a ton of submissions. Sadly I am not a designer and could not participate, but I am sure the amount of submissions would be huge.

[deleted]
(comment deleted)
loup-vaillant
Can't you control the minimum font size on your mobile browser or something?
IgorPartola
I believe on iOS you can increase the font size for the entire UI, not just the browser. Not something I would want to do. No idea about Android.
saurik
> Edit: please don't suggest one of the other HN wrappers. I know they exist.

What about one of the many apps? Have you tried news:yc?

vosper
My solution to this for my Android phone (Galaxy Nexus) was to switch to the Opera browser. Opera is one of few browsers that still supports text reflow, meaning that I can easily zoom in and have the content nicely reformatted to fit the screen. Works like a charm on HN.
pfg
See [1] on why this is happening. Relevant part:

>Internally, the web server associates a token called the fnid with each instance of a link or form, and records the continuation function for each token. Periodically, old fnids are deleted.

[1]: http://arclanguage.github.io/ref/srv.html

kijin
Thanks for the reference.

Any idea where the fnids are stored and how they are garbage-collected? Could it be that they're just dumped into a memcached instance and older fnids get pushed out whenever the memory limit is reached?

wahnfrieden
It's not the fnid that gets garbage collected, it's the coroutine itself.
gaving
I'm not sure anybody particularly cares why it is happening, just that it is happening.
jmount
I agree with you: HN time-outs and failures (which also occur on reading) are an unnecessary pain. This is one reason you should never write into a web text box (a rule I also fail to adhere to). But really even "modern application grade" web stuff is so flakey if you value your text you write it in a local editor and then copy/paste later.
BrandonMarc
With this rule in mind, I wonder if a textures / notes area would be a good feature to include in the web browser itself (including mobile browsers) ...
sergiotapia
Public Service Announcement:

If you're on Chrome use HackerNews Enhancement Suite.

https://chrome.google.com/webstore/detail/hacker-news-enhanc...

No more ass-ugly hackernews font choices and colors.

sk5t
Neat! Worth it for the option to collapse threads alone. Recently some extraordinarily lengthy, tedious threads have dominated the discussion on otherwise-interesting topics, and simply finding the end of the dreck was a problem.
cordite
Yeah, it's really annoying to have to edit the URL on mobile when this happens..
throw_away12847
I really do believe that HN is poorly designed for end users. The UI is not forward with intent, or pleasurable to use. It's been literally years and the expired link issue hasn't been addressed. ---No, I'll stop you right there. It's been acknowledged as an implementation bug, but not addressed. PG's lack of user empathy and stubbornness make HN worse.

The only reason people come here is the content.

btilly
The only reason people come here is the content.

I consider this a good thing. It attracts the right kind of people.

throw_away12847
Are you insinuating that the key to dissuading the wrong kind of people is to make Hacker News difficult to use? The notion that good users will "power through" a bad design is dubious at best.

I hope that you're not involved in product design. This thought process is not good for anyone.

btilly
I make no apologies for my opinion, however much you dislike it.

I've maintained for years that the key to a high quality online forum is for it to be limited in some way. It can be limited by bugs. Limited by nobody advertising that it exists. Limited by having a very focused topic of discussion. I'm willing to believe that it can be limited by moderation, but I self-select out of those forums so don't know.

But without some sort of limiting you get what happened to Usenet, Slashdot, kuro5hin, the main page of Reddit, and so on. Which is that they became the place that everyone knew that they should go. And then the noise went up faster than the signal, resulting in a poor signal to noise ratio. And then the best people found new forums to go to, and slowly disappeared.

Of course if you make the limiting TOO effective, eventually nobody will be left. There is a balance to be had.

And this only applies to high quality content of the kind that I want to engage with. Which is admittedly not to everyone's taste. It is not a way to make something popular.

jellicle
For any new readers: this is a "bug by design", it has been extensively complained about in the past, and it's marked "will not fix, stop complaining" by management.

This has been a public service announcement.

akkartik
Not true. It's an open bug. It's just really hard to fix.

https://github.com/HackerNews/HN/issues/11

kijin
@akkartik's JS redirect suggestion sounds like it will be trivial to implement. No need to fix the legacy arc code, just add a stopgap measure that will improve the UX for a lot of people :)
falcolas
With all due respect, if a framework makes storing sessions for more than 30 minutes "hard", there's something fundamentally wrong with the framework. This has been a solved problem for longer than I have worked in the industry.
phamilton
One positive side effect of the architecture is proper isolation in pagination. When I fetch Page 1, the link to Page 2 will always return the same set, regardless of timing (up until it expires). You will never see the last entry on one page show up as the first entry on the next page.
grey-area
I have a modest proposal for addressing the terrible dearth of fnids on this site. Since fnids are in such high demand, and yet are expensive to keep in memory, one small change might free up lots of them - remove the use of fnids for more pages and use a static url instead

so the more link on:

https://news.ycombinator.com/news

would point to:

https://news.ycombinator.com/news/2

and so on.... This would free up many fnids for use elsewhere, and also make the secondary list pages more easily cacheable and reduce load on the server.

The same could be done for other lists which currently use fnids without any appreciable change in the user experience or probably much change in the server code.

Kiro
Can someone explain why this expiration thing is happening like I'm 5? Even the most basic PHP tutorial app doesn't have this problem so I don't understand people saying it's inevitable.
BrandonMarc
So ... is this a bug or a feature?

Is the internal activity causing this unpopular behavior supposed to do this, in the first place? If the site is indeed behaving exactly as its designers want it to, then that's a rather separate debate as opposed to simply ways and means.

I'm not saying I agree with the behavior; I'm just trying to see what should be debated to start with.

Raphael
Yes, the order changes over time, so the second page will often contain items from the old first page.
dang
I'll try to respond to everything in one comment.

kogir and I talked recently about getting rid of "unknown or expired link". It's on our list. There are a lot of things on the list; we don't know yet when we'll get to it.

There's work ongoing to make HN more usable on mobile. This will likely roll out sooner than the above. We intend to roll it out incrementally, starting with the front page. We'll give people lots of notice so anyone running scrapers that are affected (like me!) will have time to change.

The look and feel of HN is unlikely to change much. We can argue about that one till the cows come home. I'm tempted to—my opinions are as strong as many of yours—but that doesn't seem like the best idea right now.

Feature requests should go to https://news.ycombinator.com/item?id=363. Software bugs should go to https://github.com/HackerNews/HN/issues. That would have been the appropriate destination for the OP, and indeed this issue has been posted about there.

Threads like the current one get lots of upvotes and comments, but they've never been on topic for HN. I'm going to bury this one now.

krapp
How is it not on topic for HN?

It would be on topic to discuss the implementation of any other site.

dang
What's on topic for HN, as the guidelines say, are stories that gratify intellectual curiosity. Meta posts gratify several things, but that is not one of them. They generate masses of comments like fast-growing weeds, because we're all full of opinions about HN, but the discussions never go very deep, or really anywhere. This has been the case for many years—pretty much since the beginning of the site—and so has the policy on it.
krapp
You don't think a community of programmers and hackers and UX nerds would or should be intellectually curious about why a forum written in a Lisp dialect works and why it doesn't, and how it solves, or doesn't solve, various problems?

I can understand why it's annoying for the staff to have to encounter these threads so often - it's annoying for users to keep running into dead links. But I don't think it's entirely true that this topic has no value to the community at large. Having opinions and discussing things is kind of what forums are supposed to be about.

SamReidHughes
But look at the actual discussion here. It's mediocre.
krapp
I guess I can't argue with that, even though I personally find the subject more interesting than politics or talk about sexism in tech or bitcoin... everyone's mileage varies.

Though I would disagree with the premise that meta-discussion, necessarily, is a bad thing. At the very least given the nature of the site and its userbase, it's going to be inevitable.

dfc 

  > I'm pretty sure that the HN community likes to read thoughtful, thoroughly
  > researched comments that are somewhat longer than one-liners.
Could not agree more and this is a great side effect of the time out. It gives the commenter yet another reason to compose their comment in a real editor. Personally I find the comment text box is too small to do a decent job of assessing and editing my comment. A real editor also makes it easy to format the above blockquote[^1] or to insert[^2] logical and consistent footnotes.[^3]

[^1]: http://www.nicemice.net/par/

[^2]: http://jblevins.org/projects/markdown-mode/

[^3]: I just wish markdown-mode would renumber the footnotes if I go back and insert a new footnote. As it is I have to pipe the output through pandoc