Why HN was down
On a comment thread, a new user had posted some replies as siblings instead of children. I posted a comment explaining how HN worked. But then I decided to just fix it for him by doing some surgery in the repl. Unfortunately I used the wrong id for one of the comments and created a loop in the comment tree; I caused an item to be its own grandchild. After which, when anyone tried to view the thread, the server would try to generate an infinitely long page. The story in question was on the frontpage, so this happened a lot.
For some reason I didn't check the comments after the surgery to see if they were in the right place. I must have been distracted by something. So I didn't notice anything was wrong till a bit later when the server seemed to be swamped.
When I tailed the logs to see what was going on, the pattern looked a lot like what happens when HN runs short of memory and starts GCing too much. Whether it was that or something else, such problems can usually be fixed by restarting HN. So that's what I did. But first, since I had been writing code that day, I pushed the latest version to the server. As long as I was going to have to restart HN, I might as well get a fresh version.
After I restarted HN, the problem was still there. So I guessed the problem must be due to something in the code I'd written that day, and tried reverting to the previous version, and restarting the server again. But the problem was still there. Then we (because by this point I'd managed to get hold of Nick Sivo, YC's hacker in residence) tried reverting to the version of HN that was on the old server, and that didn't work either. We knew that code had worked fine, so we figured the problem must be with the new server. So we tried to switch back to the old server. I don't know if Nick succeeded, because in the middle of this I gave up and went to bed.
When I woke up this morning, Rtm had HN running on the new server. The bad thread was still there, but it had been pushed off the frontpage by newer stuff. So HN as a whole wasn't dying, but there were still signs something was amiss, e.g. that /threads?id=pg didn't work, because of the comment I made on the thread with the loop in it.
Eventually Rtm noticed that the problem seemed to be related to a certain item id. When I looked at the item on disk I realized what must have happened.
So I did some more surgery in the repl, this time more carefully, and everything seems fine now.
Sorry about that.
302 comments
[ 3.2 ms ] story [ 329 ms ] threadIt's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
Don't think it's a "good of his heart situation". HN provides a benefit to YC and YC companies and attracts people to YC. As another example Fred Wilson has a very popular blog AVC and has said many times that he considers it "his secret weapon" (or something like that) because the value it provides over his competition.
I didn't mean to imply that there were. I was just curious.
> there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
Since there are multiple volunteers, I think that the site always feels important to at least one of them. I imagine that some of them have gone more than a week without giving a shit about HN, but not all of them at once. So I think there is doing things better in the future. In fact, HN keeps getting improvements behind the scenes, to keep it running, keep it interesting, and keep it from getting overrun with trolls.
It's just that we, as programmers, tend to take measures so that silly bugs do not happen anymore or that, at least, we leave big clues as to what went wrong.
In a project I had a similar issue: I was wrapping lists inside immutable lists but, due to a silly bug, I kept wrapping immutable lists inside immutable lists at every save made. So saved files would grow bigger and bigger.
And I did fix the bug and also added a big fat warning logs in case too many nested lists were detected.
pg might just as well have now added something preventing infinite recursion inside the comment tree or some WARN logging telling when a generate page is getting too big, etc.
I'd still find it very interesting to know what pg did, if any, to dodge / minimize / make it easier to determine if such an issue happens in the future.
1. Press [Home] key.
2. Read postmortem.
3. ???
Hilarious. I would have believed you if you appended "and his wallet"
And that's how processes are born.
Not necessarily. Processes are implemented by people, so they can break at any time.
The correct solution is more code, or less bad code.
I guess the lesson is to have code that alerts you about comment loops without going into an infinite loop.
Also another lesson would be to figure out a way to have better clarity into which requests are causing a timeout on the server.
Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging
Farnsworth: My God, is it really possible?
Fry: It must be possible, it's happening.
Fry: By the way, what's happening?
I love futurama more than any man could love any tv show.
This serves several purposes:
(1) It's less insane-sounding than actually talking to an inanimate object in an open work environment.
(2) It actually feels better and forces me to think more clearly when I'm talking to an actual person -- the cognitive focus is higher when the object of conversation can actually, in theory, think and talk back (YMMV).
(3) And finally, although it does require some focus on the part of the other coder, it's not nearly as taxing to them as actually helping me solve the problem or pairing up with me.
So it's a good compromise somewhere between pair programming and talking to an actual rubber duck. Again, YMMV. Maybe I'll call it "Pair Ducking."
Of course, it makes me look really good cos I just "helped" them solve their issue :)
My wife is a social worker by training, so it was pretty rare (though not unheard of) for her to be able to give me real input, but over the years I've trained her well enough to follow most of what I'm saying and nod at the right points :)
If I was a bit more clever I feel like there is a use there.
Have you tried running a debugger and stepping through the code?
Hmm, go on.
Wait a sec, can you reexplain that last bit?
You bring a detailed problem and break it down, and talk about it to someone else (who often isn't qualified to answer your questions due to knowledge/time constraints) - and in doing so - resolve the problem by challenging one's own assumptions.
This was effectively how every House episode was resolved.
1. http://blog.stackoverflow.com/2011/07/its-ok-to-ask-and-answ...
https://sites.google.com/site/codeconsultantplugin
It fascinates me how quickly I usually find the answer.
(There must be some joke involving the use of a meta-duck, but I can't come up with it. :) (Same principle applies, of course, just LISP makes the determining of "what" a bit more tricky. (insert discussion here about the general differences between debugging imperative and functional code)))
I usually just leave the question/answer online so that others can benefit for it.
Totally agree.
If you don't understand your problem, you can't make a plan. If you can't make a plan, you can't execute it.
Another interesting lesson from that book is that one should spend time on evaluation (how did this come about? Could We have fixed this sooner? How are we going to prevent it in the future?)
Ok, so this is Hacker News, it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.
You are right though, making mistakes is human as they say, and nobody dies because of this. In fact, less popularity might be good for the site's content quality. I'm just surprised by how much they care about thousands of hourly users, that what I would dream of having.
Focus on providing people what they want/need, and don't worry so much about having flawless technology until you can employ a horde of PHDs.
So, how many customers do you think he lost because of this? The answer is two, and one of them signed back up because they were impressed by the great job he did in handling the fiasco.
Moral of the story: As long as you really are making your best effort, you might be surprised how willing people are to deal with human error. Yes, they might be be mad, but a mistake is (usually) not the end of the world.
For what it's worth, I upvoted this thread specifically because we've all done something this stupid (or worse) :)
But that apart, even if there were an admin button to change the parent id of a thread, he would still have made the same mistake.
Unless the code in question was checking for loops. In that case, repl would have worked the same.
This is the kind of mistake one would make even if you were writing proper migrations. He was doing things live isn't an issue; neither is an incorrect id. The issue is the code doesn't check for loops.
One, if the data were held in a database, should a change like this be captured in the database logs? I am seeing more and more situations where I want these, I notice that they are by default turned off for mysql and wonder if this reflects a de facto judgment that logging slows performance more than is usually worthwhile.
Two, if the data were kept in a database, wouldn't something like this be prevented by a constraint preventing a comment from making itself an ancestor? But I suppose there is a slight performance hit in checking such constraints, and the case arises so rarely that this hit isn't generally worthwhile.
More generally:
I'm not a big SQL wonk anymore, but I find a lot of people have the intuition that relational databases are ill-suited for trees.
An intuition that is much closer to the truth is that almost all databases can handle trees pretty well, because there's still an unambiguous concept of ordering and containment, and you can usually arrange things so as to do range/ancestor/inclusion queries efficiently.
It's graphs with loops/without unambiguous concept of ordering/containment that are really hard.
One way involves accumulating an array of nodes already visited as the tree gets walked, checking each node as-visited for membership in the array-to-date.
The other method, a bit more of a hack, is just adding a LIMIT clause.
I think the 'WITH' clause is a great addition to the SQL standard, very much worth the learning the weirdness of its syntax and its optional 'RECURSIVE' term (which, as the Postgres documentation points out, isn't really recursion, it's iteration).
But in this case, writing a before insert/update trigger which ensure some_post.created_at < parent.created_at before setting parent.parent_id = some_post will do the trick.
I think it's more like your application is doing the logging already(probably; most of the frameworks do). If you really need it, turn it on yourself.
> Two, if the data were kept in a database, wouldn't something like this be prevented by a constraint preventing a comment from making itself an ancestor?
Copy pasting the table from another comment.
There isn't a simple check constraint you can place to ensure a parent's, or a grand-parent's, or a grand-grand-parent's parent_id isn't child.id You will have to write a trigger.This isn't really a big problem to solve. pg simply overlooked this problem. Had he not, he would have checked child.created_at > parent.created_at in his mutator method. So, when you do a post.parent = some_post(assuming mutator is parent=; replace it with post.setParent or (send post set-parent some-post) or whatever), it checks if post.created_at > some_post.created_at, and then assigns post.parent_id = some_post.id
Oh manually modifying production database on the fly ain't unheard of.
However it's still not "very Chuck Norris" on a scale of Chuck Norrisness compared to the modification of a running app directly in the REPL. I mean: it doesn't matter if you manually modify the DB itself or not when you directly modify the app from the REPL itself (the app being anyway "in charge" of the DB).
Sure, modifying manually the production DB might be an issue to some. But I can guarantee you that it's the last of your worries when you're actually modifying production code directly from the REPL ; )
I liberally sprinkle my code with assertions (CS theory calls them pre-conditions and post-conditions, iirc) to crash early if the system is an invalid state.
One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.
When I hit the first infinite loop bug on a code path, I frequently add code to assert that the number of calls is less than $A_LARGE_NUMBER to catch future occurrences of the same root cause.
The database layer often contains this logic, but it depends on how you're building your application; NoSQL backends for example typically must put validation in the application layer. Since HN just uses files, a well-developed application layer should be riddled with invariants like this.
http://www.cs.cmu.edu/~pane/research.html
http://www.cs.cmu.edu/~pane/thesis/
Pretty cool work regardless, I really like the way it deals with aggregates, for example.
Also, HN uses flat files, not database.
Might not be a bad idea, if the site were to have the two requirements "maintenance must be done on the live site from a repl" and "5 nines availability".
I am not saying this can't or shouldn't be done. I am saying a db won't directly solve it.
However, you example will work perfectly for enforcing constraints in the code via the mutator which can compare child and parent timestamps, provided pg was doing it via a mutator, and not directly changing the ids.
I'm for assertions when they are simple and don't cost much (especially during development), but it's not feasible to check every condition that should not happen.
For some reason I tend to be a fan of the "stick it in a secure box" rather than "get it right in the first place" approach..
The number of parents is almost always under 3 or 4 and never over 100. Writes occur a few times a second at peak. You are prematurely optimizing.
If the code is organized (as it should be) such that all functions which require traversal of hierarchical comments pull this from a single function, then the hash check only need be applied in that one place in the code, where it need not be visible anywhere else.
In any case I am thankful for the detailed explanation.
If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.
We've all done it. I shut down an NT4 production server because I was connected via remote desktop and clicked shutdown rather than log off. This was back in the day when there was no pop-up asking for reason you want to shut down and confirmation.
Luckily it was just our internal intranet server!
http://www.nytimes.com/2012/12/02/magazine/the-autism-advant...
I can't speak for pg, but personally, I am not going to write a migration to re-parent a single thread if the site in question is my side project, doesn't bring revenue, has some intangible benefits, but not so much that warrant putting much labor into it.
Either it would be `thread.parent = new_paret_id`; or if it occurred to me that it might introduce a loop, changing `parent=` to take loops into account followed by `thread.parent = new_parent_id`. What were you expecting? A bug tracker discussion, code commit, review, change request and deployment?
failing that you put a cap in the code that generates the page. Simple stuff.
How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!)
I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from those guys! They had discussion forum downtime."
You've obviously never worked in a for profit corporation. in such there are policies and practices put in place to prevent just this kind of newbie mistake. You never modify the live database directly. Never ever. Whether it's a bottom line property or not.
I didn't say it would negatively impact YC's business. It might make them look incompetent, but these things happen, but people don't approach YC for their website savvy, they go there for the money and the connections. Most of the VC firms i've EIR'd at have much worse IT than hn. Their sites are barely usable. It seems to just go with the territory.
Let's not be so defensive, PG can do what he likes with his site, including take it down whenever he feels like saving bandwidth. But in the real world these kinds of things get real people on a fast track to their exit interview.
Wow, really? I don't think that attitude is warranted at all.
At any company (for-profit or otherwise) there is a finite amount of time and money -- and surely we can agree that solid development/deployment practices carry an upfront time/money cost, can't we?
In an ideal world, all projects would have continuous build processes, automated tests, and management tools extensive enough to render live database surgery unnecessary.
Perhaps you've worked at companies so flush with cash that every single line of code, research project or otherwise, has gone through rigorous development/testing/deployment practices. If so, I'm jealous. I've always worked at companies that had to be choosey about how they spend their resources.
It really depends on what kind of business you're in whether this is acceptable or not.
The goal was reasonable. The action was reasonable. What are you firing somebody for? Making mistakes? Good luck making that a hiring criterion. "Ok, tell us about a time you made a mistake and what you learned from it. What's that? You never have made one? Great, you're hired!"
The solution from a retrospective should never be, "Let's make people more scared to do the right thing." Or "Let's fire people with bad luck." Firing people of PG's caliber isn't a solution, it's just another problem.
If reverting code didn't fix it, reverting server didn't fix it, incorrect data is the most likely culprit(I am not claiming this should have outright occurred to you; just thinking out loud). I take it you introduced non terminating recursion by making a thread its own parent, and you made the change on disk.
But this analysis is the last thing that comes to mind when you already have introduced 2 new variables the same day - new code, new server. And an old, recurring variable(GCing too much) is in play as well.
True hacker spirit.
(Though I have to say, upgrading the code at the same time as you're restarting to fix a problem is really a rookie mistake. It's incredibly tempting because it saves so much time, but if you do it you will get it wrong sooner or later. One of the hardest skills in programming is acquiring that zen that you need to wait in a state of readiness for the effects of your first change to make themselves apparent, rather than changing something else)
I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated.
"For some reason I didn't check the comments after the surgery to see if they were in the right place. "
More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur.
Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him?
> More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur.
How about the opposite scenario? He has done it so many times with desired results, that he didn't bother checking?
This is a big difference between engineering and hacking. An engineer would never regularly do something so dangerous.
But I suspect pg isn't an engineer when he works on HN, I suspect he is a hacker, and just does whatever he wants to, whenever he wants. Which is his prerogative.
> you run into potential problems that you hadn't anticipated.
The second statement is no reason to live by the first. In fact, I think you'd be doing yourself a disservice by staying so comfortable. Being comfortable with the unanticipated, however, is a powerful quality to have.
Obviously don't deviate from routine (or rather prescribed procedure) when you are running nuclear power plant or airplane maintenance. But when tinkering with the site that gives you no money and won't cost any lives you can loosen up a bit.
Just like too many nested if(x) { if (y) { if (z) ... }} constructs, too deep a discussion nesting is also unreadable.