Argh. And if you run it in Docker like I do, I have to relearn the lesson that taking down the container, losing internet, then trying to do a ‘docker pull’ is a a bad idea.
Sometimes I feel the attraction of some spyware mesh system for home networking.
Very high uptime overall. Credits to PG and his lisp/arc inspired codebase! And to dang (who I assume does a bit of coding on it from time to time from previous comments I have read..sorry if I read wrong)
Today it has been raining heavily, and the only reason I went to my co-working place is because I checked HN in the morning, and figured my internet was down at home because HN didn't work.
I heard from an ISP owner that a common support call was when kids deleted the browser icon on the desktop, the parent thinks internet is down and calls up support. These days almost everyone is using a phone or tablet so it happens less.
Right, HN is the site I check when I'm not sure my connection is working, because it loads so fast. It HN doesn't load, then I'm more likely to walk away from the computer than to check another site.
For decades now pings to yahoo.com to check connectivity have been my only direct interaction with them. If they go away or start responding to pings I could use news.ycombinator.com, but the URL is longer.
Browsing to HN was the first thing I tried to do this morning after accidentally sleeping in. When it failed to load, my panicked thought was: "Is this how it all ends? Did Putin finally press the red button?"
I am at gmt-5. I cannot sleep, my wife is out of town, I lost mi kindle the day before yesterday and HN was down. If I turn in the lights to read a book I will not sleep for the rest of the night so I listened to our cats fighting over what I assume was some big moth as my only distraction.
The last time we had a hard drive failure, we were down for a couple days. That was early 2013.
After that, in the spirit of 'never again', Nick (kogir) set up the failover server system that we still use today. That's why you missed it tonight; you wouldn't have back then. Sorry, I guess?
Edit: by pure coincidence, Nick was in town tonight and we met up for the first time in years. Two hours later HN goes down. almost as if the server was overcome by nostalgia
Our Diablo disk goes on the fritz, but who needs a disk when you can netboot? Ken demonstrates the Alto network capabilities, connects to Google, and has the Alto calculate and display a Mandlebrot set. Ken's in-depth blog entry including the fractal demo source code is found here:
We begin our very gentle and progressive power up of the seminal Xerox Alto. No magic smoke, but one power supply is faulty. Opening it up reveals that it had a tough life, having suffered a catastrophic short of some sort, hastily repaired, and some traces almost entirely corroded through. But the source of the malfunction seems to be a somewhat classic case of bad electrolytic capacitors, way too far gone for any hope of reforming. After replacing them and repairing the supply, we turn our attention to the Diablo disc drive and cartridge, and have a bit of a surprise.
Many thanks to my CHM restorers colleagues Ron Crane, Ken Shirriff, Carl Claunch and Luca Severini.
See previous video introducing this historically significant machine:
There is a good chance that it is (or was!) an actual spinning hard drive. Whatever it is, it lives in one of our boxes at M5 and it's in their hands for the moment.
People guess the origin of our name often. Maybe this will give you even more of a chuckle. I was not aware of the name of this computer when I named the company. https://en.m.wikipedia.org/wiki/The_Ultimate_Computer
That being said, is the actual production infra archi for HN described somewhere ? Curious how simple it can afford to be.
We laugh at people piling layers and layers and artifacts on their sites, all in the hope of adding redundancy, handle "webscale" load, and avoid an outage (ironically increasing the chances that _something_ will break).
However, if a single hard drive crashing somewhere can cause your site to be down for minutes or hours, some non-tech people (managers, shareholders, customers) will wonder if the site is "professionnal" enough - and I can sympathize with them.
> We’re recently running two machines (master and standby) at M5 Hosting. All of HN runs on a single box, nothing exotic:
CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz (3500.07-MHz K8-class CPU)
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
Mirrored SSDs for data, mirrored magnetic for logs (UFS)
We get around 4M requests a day.
Irony aside, what's the point? In theory, yes, it could work better. In practice though, HN with its two baremetal boxes has better uptime than 99,99% of the Web, including the biggest ones - just because complexity has its price.
> If you had an auto scaling kubernetes cluster with multiple redundancies using rust and 3 JS frameworks outages like this wouldn't surprise your users anymore.
No apology necessary, but I'm curious how a hard drive failure caused an outage. No RAID or mirroring? No hot spares? No clustering or distributed systems?
It was part of a mirror of identical SSDs on an LSI MegaRAID RAID card. We see occasional "spectacular" drive failures that take the machine down with a single disk failure. Usually it's just a reboot to come back up, and a disk replacement, then some hours of time to rebuild the array and get back to situation nominal.
108 comments
[ 0.74 ms ] story [ 215 ms ] threadSometimes I feel the attraction of some spyware mesh system for home networking.
Things seem to work on Firefox on mobile. Something with internet relay perhaps?
Search for more: https://hn.algolia.com/
Good that we are past the bump now
After that, in the spirit of 'never again', Nick (kogir) set up the failover server system that we still use today. That's why you missed it tonight; you wouldn't have back then. Sorry, I guess?
Edit: by pure coincidence, Nick was in town tonight and we met up for the first time in years. Two hours later HN goes down. almost as if the server was overcome by nostalgia
Sorry everyone!
It's not an actual spinning hard drive, is it?
https://en.wikipedia.org/wiki/Xerox_Alto
Diablo Systems Incorporated Series 30 Disk Drive Maintenance Manual
http://bitsavers.org/pdf/diablo/disk/model_30/81503-02_Serie...
Restoring Y Combinator's Xerox Alto, day 4: What's running on the system (righto.com):
https://news.ycombinator.com/item?id=12197591
http://www.righto.com/2016/07/restoring-y-combinators-xerox-...
Xerox Alto Restoration Part 16 - our disk goes down, the Alto connects to Google and draws fractals
https://www.youtube.com/watch?v=adEr2aRwHnI
Our Diablo disk goes on the fritz, but who needs a disk when you can netboot? Ken demonstrates the Alto network capabilities, connects to Google, and has the Alto calculate and display a Mandlebrot set. Ken's in-depth blog entry including the fractal demo source code is found here:
http://www.righto.com/2017/06/one-hour-mandelbrot-creating-f...
Xerox Alto Restoration Part 1 - power supply restoration, disk drive surprise
https://www.youtube.com/watch?v=xPyqQXFC2yw
We begin our very gentle and progressive power up of the seminal Xerox Alto. No magic smoke, but one power supply is faulty. Opening it up reveals that it had a tough life, having suffered a catastrophic short of some sort, hastily repaired, and some traces almost entirely corroded through. But the source of the malfunction seems to be a somewhat classic case of bad electrolytic capacitors, way too far gone for any hope of reforming. After replacing them and repairing the supply, we turn our attention to the Diablo disc drive and cartridge, and have a bit of a surprise.
Many thanks to my CHM restorers colleagues Ron Crane, Ken Shirriff, Carl Claunch and Luca Severini.
See previous video introducing this historically significant machine:
https://youtu.be/YupOC_6bfMI
For much more details and references, see Ken Shirriff's blog entry corresponding to this video here:
http://www.righto.com/2016/06/restoring-y-combinators-xerox-...
A 1970s disk drive that wouldn't seek: getting our Xerox Alto running again
http://www.righto.com/2018/03/a-1970s-disk-drive-that-wouldn...
Identify It Challenge for 7-26-2012 Answer
https://reinventingscience.wordpress.com/tag/diablo-systems-...
ARTIFACT DETAILS: Series 30 disk drive
https://www.computerhistory.org/collections/catalog/10266694...
Description: "Not working cards mis...
Read that as MI5 and it gave a chuckle!
We laugh at people piling layers and layers and artifacts on their sites, all in the hope of adding redundancy, handle "webscale" load, and avoid an outage (ironically increasing the chances that _something_ will break).
However, if a single hard drive crashing somewhere can cause your site to be down for minutes or hours, some non-tech people (managers, shareholders, customers) will wonder if the site is "professionnal" enough - and I can sympathize with them.
> We’re recently running two machines (master and standby) at M5 Hosting. All of HN runs on a single box, nothing exotic: CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz (3500.07-MHz K8-class CPU) FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads Mirrored SSDs for data, mirrored magnetic for logs (UFS) We get around 4M requests a day.
Good choice. :)
"It's a Unix system, I know this."
[0] https://en.wikipedia.org/wiki/Fsn_(file_manager)
(It's a wonder that anyone lets me near the frontend of their websites really.)
https://news.ycombinator.com/item?id=32024989
[0] early 90s, that is
FTFY
Does this still exist?