HN Explorer

Warning to YC applicants: We're going to restart HN tonight

103 points by pg ↗ HN
So if you're editing your YC application in the application form (which I wouldn't recommend anyway), click regularly on the Save button at the bottom of the page.

We restart HN every 5 or 6 days, or it gets slow (memory leaks). Usually we do it at night and don't say anything about it, but since people might be editing YC applications, I thought I'd better issue a warning.

I haven't said when we're going to restart it, because I don't know. It shouldn't matter anyway; just don't depend on it.

72 comments

[ 4.6 ms ] story [ 218 ms ] thread
someone13
Out of sheer curiosity, is the restart a manual thing, or is it just on a cronjob? I'm thinking by the wording of the OP, it's done manually, but I'm still going to ask :-)
pg
I do it manually.
drp4929
How do you know it has crossed slowness threshold to prompt restart ?

I check HN couple of times a day and I have not felt any performance difference.

TheMakeA
Zapier sends him a notification of course... ;)
dpe82
Out of curiosity, why not have it restart automatically every couple days at like 4am? One less thing to worry about...
duck
Then you have one more thing to worry about - whether the server/service is going to come back online. :)
smartwater
If it was on a cronjob, he would know precisely what time the server reboots.
ohlol
you assume a very primitive logic. but, given the primitive subject, i can't disagree.
blhack
I love the informality of this post.

Just "hey, guys, we're gonna flick the lights on and off sometime tonight; just a heads up in case you're working on something."

Never change, HN.

JabavuAdams
Yeah, you never really know how the sausage is made. You can watch all these videos and conference presentations of our-awesome-system-to-ensure-super-duper-scalable-uptime, spend hours debating this or that architecture, and generally be less productive than if you just kick the box in the right place, every so often.
adgar2
Yeah... writing a site as simple as HN that doesn't fall over in a week isn't rocket science.
sgdesign
This is seriously misinformed. You have no idea what goes into making a site "as simple as HN", and I very much doubt you would be able to write one in a month, let alone a week.
adgar2
Please list your assumptions about what ideas I have - since you're clearly assuming I don't work for a company that knows how to write websites that scale.

Edit: I find it adorable that you all think HN is a hard site to run. Good luck breaking 200 qps.

orangethirty
Edit: I find it adorable that you all think HN is a hard site to run. Good luck breaking 200 qps.

It is a hard site to run due to its audience. Realize that this site gets poked around by hundreds of hackers. All trying to learn or break the system. Trying to not get your ass handed to you with a commonly language/framework is hard enough. I can't fathom how they do it with an experimental language and libraries. Though they are Lisp hackers. If anyone can do it, its a Lisper...

pg
HN wasn't designed to minimize development effort. It was written to exercise an experimental programming language whose implementation is not (to put it mildly) optimized for performance. Considering its origins, it's surprising this site even has the performance it does.
adgar2
So to be clear: you care more about the integrity of exercising Arc via HN than you care about your users. Because we both know that writing HN correctly would reduce latency dramatically along with short- and long-term ops effort. It'd also fix all those nasty "dead or expired link" errors. Also, users wouldn't be terrified to click "reply" without copying their posts to the clipboard, first.
YuriNiyazov
So your first account was hellbanned, and you created a new account a month ago so that you can continue being a prick?
JabavuAdams
You raise some valid usability problems, and yet you're here commenting. So, evidently the value of the site to you outweighs your usability concerns. Meh, won't fix.
brudgers
Ostensibly, HN is a tool used to support YC's business. Though I believe that PG, et al. recognize it's importance to a broader community, it is not primarily an end in itself.

I suspect that the HN software does much more important but less obvious things which meet YC's goals than those you mention. I also suspect it does those things well.

As for page expirations, well, they aren't a deal breaker for me.

willholloway
> I suspect that the HN software does much more important but less obvious things which meet YC's goals than those you mention. I also suspect it does those things well.

If I was PG I would test YC founder's time/frequency spent on HN against the success of their startups. I would use that data in evaluating new applicants. This test might ferret out founders with a procrastination problem.

I might also try discovering possible throw away HN accounts created by founder's applying under their official HN identity. That could really ferret out some assholes not technically competent enough to cover their tracks and dumb enough not to think this possibility through.

Also, PG's essays are one of the great success stories of content marketing.

That's how I found this community, and although I probably will never apply to YC, the enthusiasm for it's backed startups have rubbed off on me. I wouldn't have been an early AirBNB or Dropbox user if not for those essays.

brudgers
My conjecture about the secret sauce is more along the lines of using statistical techniques to foster better discussions.
jamesaguilar
Maybe you should offer to pay him something to get these things fixed, if you care about it so much.
[deleted]
(comment deleted)
orangethirty
Let's also mention that other "properly engineered" sites done by "rockstar programmers" don't even come close to how well HN works. Given the traffic and the audience, it is quite a remarkable thing, actually. All this with an experimental version of Lisp. :)
sheikhimran1
true but I believe there must be a major upgrade or a CMS developed for YC only so their founders can operate from any device and more efficiently.
tylerlh
Things don't always need to be rebuilt 'just because'. We all get tired of thinking that way at some point or another.
orangethirty
They do have their own separate social network. Wonder what that runs on? I bet its Python.
sheikhimran1
Good but still the site needs a backend upgrade seeing that we are sitting in 22nd Century, there must be a major backend upgrade so you guys can work easily.
DanBC
It'd be interesting to see what you would do differently with your version of a site as simple as HN.

Do you have the code on github or somewhere?

JabavuAdams
You missed my point. When you're in a situation where you have authority and ability to rewrite anything, with a reasonable expectation of success, it's often more productive to just do anything that works, and then change it if it becomes too big a pain. It's surprising how often any extra effort / thought just isn't worth the time.
Yver
This is the part where you put your money where your mouth is and make a better HN in your favourite language. It isn't rocket science, after all. Right?
JabavuAdams
Even if that's true, consider how long it takes to type the command to reboot a server. Let's say pg reboots it 70 times a year. So, maybe he spends 1 hour to 1.5 hours a year rebooting HN. That's a better use of time than spending 2+ hours to make the site more scalable. There's no one he's interested in (i.e. deal flow) who's going to stop using HN or applying to YC because the site goes down every 5-6 days.

There may be all kinds of other reasons to update the code, but that comes down to available time, experimentation, and enthusiasm, not productivity.

Terretta
And could run it under an angel, auto rebooting when mem leak above a certain threshold and traffic below a certain level, allowing a couple days' window to look for a low impact time.

This is more or less what Apache and IIS do with child processes or long running threads. It's a production proven approach.

peterwoo
I wonder what (# of expired links clicked each year * 8 seconds / link) amounts to.
ojiikun
Memory leaks? If you're in the market for a Performance and Availability engineer with 15 years' experience in making this sort of problem a happily-fading horror story, send me a message. ;)
bengl3rt
It's written in Arc, which is written in Racket... you might find yourself coming up against the limitations of tools which were designed by academics, for academics and never meant to be used in a production site.
nicholassmith
Plus I'm sure I've seen pg say somewhere it runs reasonably well now, and when it starts going they flick the switch and bring it back up.
kenko
Does the racket-lang website have to be restarted every so often?
Locke1689
Racket is open source. Have at it. http://racket-lang.org

Based on the last time I touched it, I suspect it's a managed memory leak, though. The vast majority of Racket is in racket/base or mzscheme.

timothya
How does the restarting affect people filling out the application form? As long as the restart happens between pressing the save button, would there be any problems? The only problem that I can forsee is if the save button is pressed while the server is restarting.
JoelMcCracken
The wonders of continuation-backed web development. Have you ever had an "unknown or expired link" issue on hn? Im pretty certain they are the same problem.
thom
I suspect this isn't a full server restart, and is more just the app reclaiming memory from per-user data, like a session expiring.
robryan
At very least the process(es) running the site would be restarted.
jiggy2011
Wow, it's scary enough writing a comment in HN without a copy in my paste buffer. You would be a very brave man to do your application straight up.
pooriaazimi
Been using http://getlazarus.com/download for years - much happier!
dgesang
Wow, there really seams to be an addon for everything, even if they make a simple thing more complicated. Why don't you just copy your post before you hit the 'submit' button? Easier, faster, better?!

FWIW: Use Ditto if you don't want to lose the previous buffer content.

pooriaazimi
> Why don't you just copy your post before you hit the 'submit' button?

Three simple (pragmatic) reasons:

1. Most of the time I forget.

2. Sometimes I'm writing multiple comments in HN and elsewhere simultaneously. And, sometimes I write quite long, carefully thought out comment on a forum I visit frequently, complete with images and all that (the forum runs phpBB and doesn't have a 'save draft' button). We need to change the web server before we can move to a better forum engine (vanilla, perhaps) and it's a small forum and doesn't worth the effort until we move the forum to another server later this year.

3. I usually use Safari betas (on OS X), and they're not famous for being stable and while the whole app rarely crashes, after they've moved to WebKit 2.0, occasionally something goes wrong and it has to refresh all open tabs and flush anything that's been in the text fields. So, even if I do #1, the danger is still there.

:)

JoshTriplett
Better yet, copy the application questions to a text file, write your answers there, and when you get ready to submit your application, copy in the answers and hit "submit".

You'll want to preserve a copy of your application for future reference anyway.

tokenadult
Better yet, copy the application questions to a text file, write your answers there, and when you get ready to submit your application, copy in the answers and hit "submit".

When HN gets busy, that is also helpful for posting comments. Sometimes even when I type one of my shorter comments, by the time the comment is done and I select "reply," the response is "unknown link," a sign of a time-out condition. (I probably type slower than many programmers.) Fortunately, my browser preserves my form submission if I encounter that condition, so then I copy the whole comment and try again. Anything long that takes looking things up I compose in my text editor, and then paste in all at once (as for another reply I posted today).

[deleted]
(comment deleted)
adgar2
> We restart HN every 5 or 6 days, or it gets slow (memory leaks).

Because competence wasn't worth it a few hundred restarts ago and isn't worth it now.

yozmsn
Because trolls are the best way to make people laugh.
viralbajaria
aahh!! now i know why my application submit button would work sometimes even after i had the browser open for a few hours. Now I know that whenever it didn't work I could have been a victim of this restart.

thanks for the info.

zheng
I'm not going to lie, I had a good laugh thanks to this post. Not that I fault the method being used to combat memory leaks. Depending on how much (of pg's) time a restart takes, this may be the most efficient way of handling it, but the idea that the site where we all discuss the finer points of multi-region-zone-datacenter replication redundancy just gets restarted when something isn't working correctly humors me.
andrewljohnson
Pretty sure the most efficient way of handling it would be to port the site to something that wasn't Arc. These problems exist because Hacker News is half science project, half production site.
bane
The best part is that it's becoming one of the more influential tech sites on the web now, kind of proving the point that hacking together something great (and NOW) can provide tremendous value to lots of people.
sharkweek
I am a diehard HN fanboy, love the community, love getting my daily tech news here; but... anytime I want a good laugh I like to crack open the source code of Ycombinator's home page

I mean... c'mon

<table width=100% cellpadding=6 cellspacing=0> <tr><td bgcolor=#ffff88> <table width=100% cellpadding=0 cellspacing=0> <tr><td bgcolor=#ffff88>

yuhong
Yet the JS still uses getElementById!
bane
It's even funnier when I think that HN is probably more popular than anything I'll ever work on. Apparently people don't care that you are using the latest dynamic stylesheet code generator or whatever, so long as the information is presentable and the community is good.
tripzilch
There's plenty of people that care, but it seems like it's not going to be fixed anyhow, so you don't see many complaints.

Personally I'd love it if the HTML template of HN would be a bit more semantic. I've had at least four false starts where I was like "yeah I'm gonna build this piece of userscript JS so I can more easily <check replies to my comments or whatever>" only to hit a brick wall when I am reminded how awful it is to parse, even with today's advanced DOM querying features. Fortunately there are those with more perseverance, so I am thankful to at least get to use someone's bookmarklet that adds comment-folding to HN :)

ohlol
dat computer science dat lisp dat dat
Snapps
Social news sharing FTW! I love how we can help others get this news simply through our engagement.

Oh, and as for "editing your YC application in the application form" ...I wouldn't recommend it, either. Use something that doesn't require an internet connection, and save often. [Don't forget to actually fill out and submit your application, though. =]

SeoxyS
I'm curious, does the use of continuations prevent HN from running multiple processes & from being load-balanced?
irahul
> does the use of continuations prevent HN from running multiple processes & from being load-balanced?

AFAIK HN doesn't use continuations, but closures.

Something along the lines of(imitation of arc challenge in flask):

https://gist.github.com/3998745

kjhughes
If you restart while I'm in the middle of describing how my start-up simultaneously solves every problem from your frighteningly ambitious post, know now that I will NOT be retyping it.
uuilly
What time are the applications due? It just says Nov 2. Should I assume just after 11:59 Nov 1st?
etfb
So hang on... you're telling me there are people who consider themselves geeks who don't use the Lazarus plugin (http://getlazarus.com/) in Firefox? (Or an equivalent, but Lazarus was developed by a friend of mine, so I prefer it.) That's just weird.
chacham15
Im sorry for my ignorance, but I dont understand how HN can have memory leaks. I learned Scheme in school and from what I remember memory is handled automatically (in the particular one we were studying I think it was via reference counting). Is Arc that different from Scheme or am I just completely unaware of how Lisp works behind the scenes?
philwelch
Garbage collection is cruise control for memory management, but you still have to steer. The only thing GC gives you is automated cleanup of orphaned data; if you're sloppy about holding references you don't need anymore, the GC won't help you.