The recent iOS update changed the default appearance of buttons in Safari, and not many sites use the raw button appearance, but HN does. So that could be what you're seeing
Can't tell, but logging out to get the cached pages did seem to help... so maybe it's a lot of people all logged in hitting the same database each click?
I rather suspect that it's because of the Facebook outage, but indirectly. HN is always slow when major tech services (Facebook, Google, AWS, Github, Slack, Fastly, Cloudflare, etc) services are down. The HN community seems to congregate here for updates causing higher than usual traffic.
DNS lookups are certainly slower for random sites. I just opened an about zero traffic site that i hadn't gone to in a while (so dns not cached). It took a while (10 seconds?) to load the first time. Insta load the second time.
This seems to happen for .com but not other TLDs so maybe .com is getting hammered?
This is definitely the correct take. I fall into the 18-24 demographic and I don't think I've used Facebook at all in the past 3-4 years. Neither does anyone else I know who is my age.
This is essentially a reverse network effect: because so few young people are on the platform, few other young people see a reason to use it.
Most Facebook users don't know what HN is, but most HN users have Facebook accounts and will migrate here for the dopamine-inducing refresh clicks (don't let them try to claim otherwise ;) ).
If most Facebook users knew what HN was and congregated here when FB was down, the site would be utterly crushed.
Either there is a massive number of HN users F5ing the homepage for the last few hours, HN is woefully unprepared for surges in traffic, or there is some kind of outage with their infrastructure.
hnstatus on twitter and hund.io show no outage news.
It'll be this, I'd bet money on it. Lots like me who don't visit every day are coming to HN for more info. This site probably only ever sees a tiny percentage of its registered users accessing at once; right now, it'll be a much larger percent.
IIRC HN runs on a single server. So they don't really put much effort into scaling to higher traffic levels. They don't really need to. Even now the site is available, if a little slow.
Telegram is buckling under this too. Only major comms for me not slowing down at the moment are Fastmail (email) and Matrix (which we have the homeserver on our own ASN)
Seems like the opposite lesson to me - Telegram is kind of iffy, not sure about the others. The only thing that's been really solid today is Google. So I guess depend on multiple enormous centralized services?
The underlying problem is that HN is near the limit of its hardware all the time, which could happen on one server or on 200 servers. And the reason it's near the limit is because it's single-threaded, I believe. Fix it to use multiple threads, and one centralized server will handle load spikes like this just fine.
Nginx is single-threaded. At least was when I last looked.
Node.js similarly is single-threaded. Both boast pretty impressive performance.
Being single-threaded is not a detriment to performance, rather, it most often results in faster software, per core, due to lack of the need for synchronization and avoiding a memory allocation that happens for each thread (can be a problem, see C10K problem).
Being single-threaded does not mean you can't do things concurrently. As long as you have a set of async primitives, starting with the venerable select(2), and a bit of language syntactic sugar (preferably, or just setjmp(3)) you can interleave handling of incoming and outgoing data in what is called "cooperative multitasking". There are many higher level frameworks built on top of the principle, which is that - hidden or visible - context switches are voluntary only. As HN seems to be written in Arc, which shares the implementation with Racket, then the feature most likely to be used for this is either call/cc (or one of the higher-level abstractions built upon it, like generators) or it would, going past the "cooperative" territory, simply use threads: https://docs.racket-lang.org/reference/threads.html These threads are not "real" threads, only one instruction among all threads ever executes at once, but the points of context switching are not specified in the code being executed (leading to preemptive multitasking but without parallelism). If HN uses those, then it can be called both "single-threaded" (the VM) and "multi-threaded" (the HN code) at the same time.
Being single-threaded also doesn't prevent you from using multiple cores to max out your processing power. In age old tradition you can just spawn off a few worker processes and call it a day. You might need some sort of Inter-Process Communication, but there are many mechanisms to choose from. Nowadays on Linux you can use the listening socket as a simple round-robin scheduler for processes, free of charge, no need for a separate proxy. The processes might be managed by external supervisor, like systemd, or they might be managed by the language runtime itself, and Racket provides Places for this IIRC: https://docs.racket-lang.org/reference/places.html
Note: I'm just guessing based on what I've heard about HN implementation and my knowledge of Racket which may, or may not, be relevant at this point.
Anyway: rewriting the HN backend to "use multiple OS-level, preemptively scheduled threads" (which you wanted to write but contracted to "multiple threads"), would not result in per-core performance increase, it's actually more probable to cause the exact opposite. I'd guess that the backend is already optimized in terms of algorithms and data storage, so the only recourse now would be to rewrite the whole thing in (or compile down to) a language that tends to perform better than what Racket VM can offer. Given that HN is already a weekend project of a hacker (I mean, written in a Lisp? A Lisp implemented on (former) Scheme...), I'd probably give Nginx + LuaJIT a whirl, or maybe one of the BEAM languages (there's a Lisp over there, too, so another fun weekend for porting Arc to LFE) if manual control over distribution over multiple nodes is desirable (but it would be a total time-sink, while things like Kubernetes (I'm told) mostly work well enough... though K8s is a time-sink in its own right...)
Dang mentioned in another comment the GC being too slow. In that case, replacing the GC implementation could help, if possible. It's also possible to write code that avoids as many allocations as possible, but it's hard, and the GC can still happen at bad moments....
That's a lot of words to complain that I didn't explicitly say "and single-process".
My statement isn't "very wrong" at all. If you actually have one single thread, total, that's a harsh limit on how much you can do. And I never implied it wasn't concurrent within that thread.
That's still wrong: scheduling and synchronisation take time, especially synchronization, which has both an overhead and introduces uncertainty in when the lock will be released. As I said, multi-threading results in worse per core performance, and depending on your data it might actually even perform worse on multiple cores (sometimes; most often it offers a moderate speed-up, not what you'd expect).
You also presented the threadedness as a cure-all solution to performance problems, which it is not. That's wrong.
You also never implied that you mean paralelism in general as opposed to just threads. But yeah, in the case you can't use any kind of parallelism, you're kind of screwed. The problem is that threads are one of the worst ways of introducing parallelism.
> That's still wrong: scheduling and synchronisation take time, especially synchronization, which has both an overhead and introduces uncertainty in when the lock will be released.
No matter your method, you need some kind of scheduling and synchronization if you want to go beyond 5 billion CPU cycles per second on a multi-user site.
> You also presented the threadedness as a cure-all solution to performance problems, which it is not. That's wrong.
No I didn't. But having more than one core in use is the only way to get over a certain threshold, no matter how good your code is.
> The problem is that threads are one of the worst ways of introducing parallelism.
Like I said, it would have been better if I was clearer that I mean completely non-parallel code. But parallelism was my point. Everything you're saying about threads in a single process vs. multiple processes is not me being wrong, because it wasn't my argument.
Or to put it another way: Yes threads in a single process are on the bad end of parallelism. But they're also the easiest. If you see a statement that code can't even do that, then that should already imply that you can't run multiple processes on the same website instance.
Ok, let's chalk it up as a miscommunication for the most part, I'm sorry for calling your comment "very wrong". We can agree that it was incomplete/imprecise, right? It got explained, so let's move on :) But, I take issue with one more bit here:
> But they're also the easiest.
I don't believe that's true. Threads are the easiest to work with if you have a lot of shared, mutable state kept within an adress space of a single process and, crucially, cannot do anything about it. I don't know if that's the case for HN backend, but it rarely is in practice. Then there's a problem of multiple different threading implementations out there, and also an issue of how the threading mechanisms interact with the runtime (Python is multithreaded, but has a GIL). Frankly, it's a minefield on all sides, and implementing a system that is multithreaded, performant, and correct at the same time is the exact opposite of what I'd call "easy". That's also the reason why I disagree here:
> If you see a statement that code can't even do that, then that should already imply that you can't run multiple processes
To me, threads are not the first thing that comes to mind when I think about parallelism. Being unable to leverage threads, to me, is a good thing: just let this sleeping can of worms lie. As you know, there are multiple (many) ways of achieving parallelism, and they differ very much: I don't think being unable to leverage one of them says anything about the ability to use any of the others. That may be just me, though, so let's just say I have a different impression here and call it a day :)
Sure, very little to disagree with, and moving on is fine, but I do want to make one small note: the reason I say threads are easier is that in almost every case you can take a multi-process model and turn it into an efficient multi-thread model with minimal effort. Threads give you more tools, and a lot of those tools are slow. And if a type of parallelism won't work with threads, it very likely won't work with anything else.
Apparently DNS services are getting hammered with much higher traffic than usual for Facebook domains (as FB DNS is down so millions of devices are continually retrying the lookup).
For twitter specifically, I also wouldn't be surprised if a non-trivial number of people have increased their use of twitter while facebook is down.
Running in anonymous mode (clear browser cache) makes it responsive, but read-only.
Edit: I suggest using a private tab or two browsers or two profiles, one where you are logged out to read comments for speed, and if you wish to comment or vote then paste a comment url over to the other. Side effect: you improve responsiveness for everyone!
When logged in the site shows you for each entry (post/comment) whether you voted on it or not, thus it has to request a store with per user unique data. (Also custom color, your points, ...) When logged out all get the exact same data.
Yes, that's normal. If you're not logged in, HN serves you a cached version that is a couple dozen seconds old. That's super fast. If you're logged in, HN has to generate the page based on the previous actions and properties of your account (e.g. upvotes, favourites, etc.) with every HTTP request.
It's just people flocking to HN for reliable news on the insane outage affecting global Facebook services, and refreshing far more than they usually would. Data point of one
"People look for alternatives and want to know more or discuss what’s going on. When Facebook became unreachable, we started seeing increased DNS queries to Twitter, Signal and other messaging and social media platforms." from Cloudflare via https://news.ycombinator.com/item?id=28752131
> General tip: If HN is being laggy and you're determined you want to waste some time here, open it in a private window. HN works extremely quickly if it doesn't know who you are.
Saw this from a user by the name of bentcorner in the "Facebook-owned sites are down" thread[0]. Figured I'd repost it here. It works because HN can cache pages more efficiently when it doesn't have to include user data.
It's facebook outage rubber necking. Everyone who has been involved with keeping services online is like "oh heads are going to fucking roll today, let's go see what folks are saying happened". haha
It actually feels like a periodical DNS outage. On Friday some of my servers weren’t responding like they should. Half an hour later everything was fine again. However, some VMs running in Azure Cloud were still having problems. Today this huge FB/WA/Instagram.
I bet we will read more about this anytime soon in the news.
It was my perception 10 years ago that HN was powered by a single threaded process with the entire post history loaded into memory (not sure how accurate this perception was). I am curious how much it has changed.
The original comment in the thread has that information, and it's nothing to write home about. Maybe it has improved since, but it's not like they're running a network of supercomputers.
So...maybe HN has dependencies on FB (such as tracking pixel, etc.)...and with their outage, timeouts kill things for HN? Not blaming HN, just wondering if second order diminished experiences start happening because of FB's heavy gravitational pull?
I can't load user profiles at the moment, and some pageloads are just spinning wheels. Not sure if that is related to network issues or if just a lot of us are rushing to HN to check/comment
EDIT: We're having some trouble serving your request. Sorry!
158 comments
[ 2.8 ms ] story [ 229 ms ] threadSeems fb down is making people go other places meanwhile maybe_
Normally, very fast for me.
IMO it's probably a lot of us tech workers watching in awe of what's going on when a big tech company craps itself.
That being said, I've seen theories that certain non-FB sites are slow just because DNS lookups are so backed up.
I have been getting a Hacker News error message more often then usual today. So I don't think it is just DNS.
This seems to happen for .com but not other TLDs so maybe .com is getting hammered?
Actually, my experience with FB (I have an account, but don't spend much time, there), is that it is packed with grayhairs, like me.
I think the younger folks have other venues they frequent.
This is essentially a reverse network effect: because so few young people are on the platform, few other young people see a reason to use it.
We like to throw anyone over 40 into the skip (especially in the tech industry).
It has caused some issues.
The company does have a fairly young, highly-paid, workforce.
I assumed the user base.
If most Facebook users knew what HN was and congregated here when FB was down, the site would be utterly crushed.
hnstatus on twitter and hund.io show no outage news.
It'll be this, I'd bet money on it. Lots like me who don't visit every day are coming to HN for more info. This site probably only ever sees a tiny percentage of its registered users accessing at once; right now, it'll be a much larger percent.
Node.js similarly is single-threaded. Both boast pretty impressive performance.
Being single-threaded is not a detriment to performance, rather, it most often results in faster software, per core, due to lack of the need for synchronization and avoiding a memory allocation that happens for each thread (can be a problem, see C10K problem).
Being single-threaded does not mean you can't do things concurrently. As long as you have a set of async primitives, starting with the venerable select(2), and a bit of language syntactic sugar (preferably, or just setjmp(3)) you can interleave handling of incoming and outgoing data in what is called "cooperative multitasking". There are many higher level frameworks built on top of the principle, which is that - hidden or visible - context switches are voluntary only. As HN seems to be written in Arc, which shares the implementation with Racket, then the feature most likely to be used for this is either call/cc (or one of the higher-level abstractions built upon it, like generators) or it would, going past the "cooperative" territory, simply use threads: https://docs.racket-lang.org/reference/threads.html These threads are not "real" threads, only one instruction among all threads ever executes at once, but the points of context switching are not specified in the code being executed (leading to preemptive multitasking but without parallelism). If HN uses those, then it can be called both "single-threaded" (the VM) and "multi-threaded" (the HN code) at the same time.
Being single-threaded also doesn't prevent you from using multiple cores to max out your processing power. In age old tradition you can just spawn off a few worker processes and call it a day. You might need some sort of Inter-Process Communication, but there are many mechanisms to choose from. Nowadays on Linux you can use the listening socket as a simple round-robin scheduler for processes, free of charge, no need for a separate proxy. The processes might be managed by external supervisor, like systemd, or they might be managed by the language runtime itself, and Racket provides Places for this IIRC: https://docs.racket-lang.org/reference/places.html
Note: I'm just guessing based on what I've heard about HN implementation and my knowledge of Racket which may, or may not, be relevant at this point.
Anyway: rewriting the HN backend to "use multiple OS-level, preemptively scheduled threads" (which you wanted to write but contracted to "multiple threads"), would not result in per-core performance increase, it's actually more probable to cause the exact opposite. I'd guess that the backend is already optimized in terms of algorithms and data storage, so the only recourse now would be to rewrite the whole thing in (or compile down to) a language that tends to perform better than what Racket VM can offer. Given that HN is already a weekend project of a hacker (I mean, written in a Lisp? A Lisp implemented on (former) Scheme...), I'd probably give Nginx + LuaJIT a whirl, or maybe one of the BEAM languages (there's a Lisp over there, too, so another fun weekend for porting Arc to LFE) if manual control over distribution over multiple nodes is desirable (but it would be a total time-sink, while things like Kubernetes (I'm told) mostly work well enough... though K8s is a time-sink in its own right...)
Dang mentioned in another comment the GC being too slow. In that case, replacing the GC implementation could help, if possible. It's also possible to write code that avoids as many allocations as possible, but it's hard, and the GC can still happen at bad moments....
My statement isn't "very wrong" at all. If you actually have one single thread, total, that's a harsh limit on how much you can do. And I never implied it wasn't concurrent within that thread.
You also presented the threadedness as a cure-all solution to performance problems, which it is not. That's wrong.
You also never implied that you mean paralelism in general as opposed to just threads. But yeah, in the case you can't use any kind of parallelism, you're kind of screwed. The problem is that threads are one of the worst ways of introducing parallelism.
No matter your method, you need some kind of scheduling and synchronization if you want to go beyond 5 billion CPU cycles per second on a multi-user site.
> You also presented the threadedness as a cure-all solution to performance problems, which it is not. That's wrong.
No I didn't. But having more than one core in use is the only way to get over a certain threshold, no matter how good your code is.
> The problem is that threads are one of the worst ways of introducing parallelism.
Like I said, it would have been better if I was clearer that I mean completely non-parallel code. But parallelism was my point. Everything you're saying about threads in a single process vs. multiple processes is not me being wrong, because it wasn't my argument.
Or to put it another way: Yes threads in a single process are on the bad end of parallelism. But they're also the easiest. If you see a statement that code can't even do that, then that should already imply that you can't run multiple processes on the same website instance.
> But they're also the easiest.
I don't believe that's true. Threads are the easiest to work with if you have a lot of shared, mutable state kept within an adress space of a single process and, crucially, cannot do anything about it. I don't know if that's the case for HN backend, but it rarely is in practice. Then there's a problem of multiple different threading implementations out there, and also an issue of how the threading mechanisms interact with the runtime (Python is multithreaded, but has a GIL). Frankly, it's a minefield on all sides, and implementing a system that is multithreaded, performant, and correct at the same time is the exact opposite of what I'd call "easy". That's also the reason why I disagree here:
> If you see a statement that code can't even do that, then that should already imply that you can't run multiple processes
To me, threads are not the first thing that comes to mind when I think about parallelism. Being unable to leverage threads, to me, is a good thing: just let this sleeping can of worms lie. As you know, there are multiple (many) ways of achieving parallelism, and they differ very much: I don't think being unable to leverage one of them says anything about the ability to use any of the others. That may be just me, though, so let's just say I have a different impression here and call it a day :)
For twitter specifically, I also wouldn't be surprised if a non-trivial number of people have increased their use of twitter while facebook is down.
https://twitter.com/Twitter/status/1445078208190291973
> Twitter starts to fail under load
I think it's only natural, given they're the biggest social networks. I'd guess they have significant overlap.
If you want to read HN, logout, or use the private mode
Running in anonymous mode (clear browser cache) makes it responsive, but read-only.
Edit: I suggest using a private tab or two browsers or two profiles, one where you are logged out to read comments for speed, and if you wish to comment or vote then paste a comment url over to the other. Side effect: you improve responsiveness for everyone!
(along with the rest of the world)
Unlike one of the other comments, hn is slow relatively rarely (perhaps a few times a year).
Saw this from a user by the name of bentcorner in the "Facebook-owned sites are down" thread[0]. Figured I'd repost it here. It works because HN can cache pages more efficiently when it doesn't have to include user data.
[0] https://news.ycombinator.com/item?id=28750513
I bet we will read more about this anytime soon in the news.
(meaning if HN is down you can't post about that, and if it's not down you won't post it; proving that you can never have a post "HN is down" on HN)
edit: found this https://news.ycombinator.com/item?id=16076041
Edit: but we can't GC it fast enough. On a day like today I wonder if always going to disk mightn't be better.
Do you (or anyone else) have an article on HN's infrastructure? Would like to read more about this.
Tried to Google it but was filled with links to other people's infra on HN.
EDIT: We're having some trouble serving your request. Sorry!