Why am I writing a data store? Because my online backup service needs to store 200,000 key-value pairs for every $/month of revenue, and existing data stores aren't optimized very well for very large numbers of very small records.
Really? Every one of the dozens of open-source key/value stores is so far off from your requirements that they just don't work for you?
Yes. The most common problem is being designed for large values. My situation of having a large number of very small values is unusual, so it's not surprising that existing code isn't optimized for this case.
Did you know that only two, maybe three of those datastores are any good? And did you know that of those three, they're not designed for every workload?
If it is not too much to ask, what are your real constraints and requirements and what existing systems did you investigate?
Based on the tidbits you've supplied I would guess what you're really doing writing is a file system capable of handling small files well (i.e. something along the lines of ReiserFS).
Tarsnap data is stored in S3; my problem is handling the metadata which allows the Tarsnap service to find user data on request. The two large tables have (key, value) lengths respectively of (41, 12) and (8, 16).
I need immediate durability; fast restart; high throughput bulk writes; reasonable throughput random reads; range requests; and (for financial reasons) I'd like to have a high disk:RAM ratio.
"Shunya" - sanskrit word for zero/null/nothingness
it also signifies existence(infinity/everything) is contained in nothingness.
should work as a gud koan too :P
33 comments
[ 2.5 ms ] story [ 83.9 ms ] threadReally? Every one of the dozens of open-source key/value stores is so far off from your requirements that they just don't work for you?
The datatorium
Ferret (no reason, it just sounds right)
Niblets (because it stores small chunks)
Petadata / petapairs (because it works with many k/v pairs)
Basology ((data)base + ology)
Keyology (the -ology of keys)
Tupology (because a k/v pair is a tuple)
SandCastle
SandStore
Sandy
SnipStore
TinyOcean
YAYSP (Yet Another Yak Shaving Project)
Actually, I am pretty sure the name CouchDB was inspired by REST :)
Iron Molehill
Gidget
Kahuna
Hailstorm
Ark
It's the Finnish word for "size" or "bulk" and does not have many Google hits (always an advantage).
Feel free to use it :)
And a cluster of Tar Pits = La Brea (which is actually a cluster of tar pits)
sounds better than "vancouver cabinet" and is technically more accurate
smallKeys
even tho you're referring to small values, "smallKeys" sounds better than "smallValues" (tho misleading)
Based on the tidbits you've supplied I would guess what you're really doing writing is a file system capable of handling small files well (i.e. something along the lines of ReiserFS).
I need immediate durability; fast restart; high throughput bulk writes; reasonable throughput random reads; range requests; and (for financial reasons) I'd like to have a high disk:RAM ratio.
Sauron
Kobold