It surprises me that Tokyo Cabinet is so slow to load, as I've had great success with it on my own, but I've not used Tyrant/pytyrant. I'd love to see some information on where the bottlenecks are in the loading process, as it seems to me that there's something funky going on that can likely be improved pretty easily.
Edit: He notes in the comments that a switch to pytc (direct Tokyo Cabinet bindings rather than going through Tyrant) would improve things, but that he wanted to keep it distributed. I'd still like to see where the bottlenecks are, though.
wow how come the bubble hasn't burst yet on key-value dbs?
first: unless you run google, yahoo, ebay, amazon, or another top-100 site, chances are you can stretch mysql/memcache more than far enough for your needs. if you do run google or yahoo, you've been using proprietary key-value dbs for a decade, you don't need to read about them on highscalability.
second: they don't do anything. forget about practically everything that makes you cozy using your open source rdbms. key-value stores don't have standard query langs, they don't have DBI-like language abstractions, etc etc. you have a key, you get a value. fun! they're fast because they don't do anything
third, the key-value systems that are rock-solid are proprietary and tied very much to large serving architectures. the open systems are half-baked at best
most hackers don't need a "scalable" site, they need a site that people actually give a crap about. build one then you can scale it
There I was brainstorming ideas to replicate the hierarchical database/collection/document management that MongoDB and CouchDB offer. One solution I have used, and became disenchanted with, was combined keys with all this information munged together. It would be interesting to hear what others do with their key/value stores.
6 comments
[ 10.1 ms ] story [ 219 ms ] threadEdit: He notes in the comments that a switch to pytc (direct Tokyo Cabinet bindings rather than going through Tyrant) would improve things, but that he wanted to keep it distributed. I'd still like to see where the bottlenecks are, though.
ttserver test.tcb#opts=ld#bnum=1000000#lcnum=10000
d specifies a deflate encoding. The type of database may also make a difference on loading times; in this case it was a B+ tree.
first: unless you run google, yahoo, ebay, amazon, or another top-100 site, chances are you can stretch mysql/memcache more than far enough for your needs. if you do run google or yahoo, you've been using proprietary key-value dbs for a decade, you don't need to read about them on highscalability.
second: they don't do anything. forget about practically everything that makes you cozy using your open source rdbms. key-value stores don't have standard query langs, they don't have DBI-like language abstractions, etc etc. you have a key, you get a value. fun! they're fast because they don't do anything
third, the key-value systems that are rock-solid are proprietary and tied very much to large serving architectures. the open systems are half-baked at best
most hackers don't need a "scalable" site, they need a site that people actually give a crap about. build one then you can scale it
OTOH, he can do that in any rdbms because he does not have large number of simultenous users.