Ask HN: How to build something like redis purely in Node.js

I would probably start by looking into hashing and sharding algorithms.... with memcached, this is done in the client, but you could just as well enable a gossip-like protocol where the servers act as a relay client to other servers... or do it at the client. In any case, if you look at some of the JS clients for memcached, you can see how they are doing distribution...

Beyond that, for non-disk persistent keys, you can simple use an object in memory, and have the keys be a property on that object. Avoid delete, set a key to `undefined` on a delete command, you should probably expose the garbage collector and force a collect on each delete operation, to keep GC cycles small, and try to reduce memory (more on that below). you can do simple disk persistence every N seconds via JSON.stringify (note, you may want to partition objects in memory so that the persistence goes faster).

You may also want to look at some of the ZeroMQ bindings, as that could work for your communication layering. Beyond that you may want to have a separate shard key from your record key... this can keep multiple keys on the same server, and reduce lookup times for related data, if you implement such a thing.

It's worth noting that your whole process with die a fiery death if you hit more than a couple GB of memory usage... node is a really bad environment for this type of work under any significant load. Array searching, btrees, and the like won't actually do any better in practice, it seems to be related to some kind of reference handle allocation.

You can also use fork/cluster to run a localized cluster to minimize the per-instance memory footprint.

Good luck, sounds like a fun project.