What caused that outage
The News server currently crashes a couple times a day when it runs out of memory. All the comments and stories no longer fit in the 2 GB we can get on a 32 bit machine. We'd been planning to upgrade to a new 64 bit server. In the meantime it was arguably a rather slow form of GC.
Unfortunately the process somehow got wedged in the middle of segfaulting. We're not sure why and will probably never know. But that meant the process that usually notices when News is wedged and restarts it was unable to kill it.
Fortunately rebooting the machine solved the problem. But now we'll presumably be switching to that new, bigger server sooner rather than later.
As far as I can tell it was a coincidence that this happened today. It doesn't seem to have been caused either by the increased traffic, or the excessive number of posts about Erlang.
64 comments
[ 6.4 ms ] story [ 108 ms ] threadAnd I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language.
It's great to see people eat their own dogfood.
That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.
I don't want to speak for pg, but I highly doubt this. He chose the architecture he did for good reason. Looking things up in memory is much faster than looking them up in the database. The only issue is that the site is too big to fit entirely into memory now, so he needs to write something that cleanly manages the memory space. (BerkeleyDB calls this a pager; data is stored in pages that can be in memory or disk.)
With that in place, the site should perform fine.
Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone to use Rails. Rails is not the only way to do things, and it's rarely the best way.
I don't think your "poor fit" comment is true. You don't need to model the threaded nature of comments at all in your database. Since there's a relatively small number of comments per post, and most access to comments is probably show-me-all-comments-for-this-submission, you usually just need a single foreign key, each comment to its original grandparent submission.
In this specific context the relevant advantage of SQL over rolling your own is that there is less distinction between keeping data in memory and on disk. Otherwise, you can certainly do anything with memory + SQL that you can with memory + disk.
If you use an object database, you can store your data exactly as it is represented in memory. That is much cleaner, IMO.
(Relational databases are like programming languages. Just like you can use any programming language for any task, and you can hack any data you want into the relational model. But that's not always the best way. Sometimes a key/value store, or an object store, or a document store is a better model. Using a better model means you need to write less code, which means your app will have fewer bugs.)
In my experience, 64bit isn't a good way to save memory.
I thought it was just a sign that I should get back to work, but maybe my account got put into some sort of inconsistent state?
Here's a page with some more details: http://news.ycombinator.com/item?id=452005
It's common for dynamic languages to embed typing information in pointers as an optimization. For example, CLISP uses at least 2 bits to distinguish between common types. That way fixnum numbers can be recognized and added without slow memory accesses.
The result is that you get less bits for the address. Hence less addressable memory.
For example, if you have an string type, which keeps track of it's length, then you might need 8 bytes. 4 for the pointer to the string of chars, 4 for the integer to keep count.
Here's a better example from tinyscheme:
As a minimum, each object takes up max(sizeof(_string), sizeof(num), sizeof(port), sizeof(_cons), sizeof(foreign_func)); And num is defined as follows:I presume that you're not hitting a melt-down from CPU time, so I'd assume you'd get better performance by letting the system handle paging in data from the disk and figuring out which stuff needs to be in memory (buffers) and lives out on disk.
Also one thing to watch out for when you make the 64-bit jump is that (internally pointer-heavy) applications in dynamic languages tend to use significantly more memory on 64-bit platforms. davidw talked about this some here:
http://journal.dedasys.com/2008/11/24/slicehost-vs-linode
You're storing EVERYTHING in memory? Isn't this kind of.. well, stupid, frankly.
A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.
You could always use Erlang...
database are not tied to a storage medium. there is no reason why you can't run a DB (key-val or even a full RDBMS) in RAM.
http://god.rubyforge.org/
(if that's the issue)
Which I believe should be called "lightning", right?
(I can imagine an interface that is transparent at the arc level, where are strings are just passed to the backend and retrieved from it, and the backend converts them to and from byte strings. Later on it could change to use a FS or a DB or whatever.)