HN Explorer

OpenKeyval.org, a free data store/social experiment

46 points by markchristian ↗ HN
Howdy, HN'ers; A few weeks ago, we had an idea for a fun side project: a sorta-Memcache-like data storage system that could be trivially used for building simple web apps. After a few evenings of hacking, @dustball and I are pleased to present OpenKeyval.

OpenKeyval is a completely open key-value data store, exposed as a drop-dead simple web service. The goal is to make this a very easy way to persist data in web applications.

http://openkeyval.org/

As a data storage system, it's pretty basic: you can store any data you want under a particular key, and then fetch it later. You can also serve it back with a particular MIME type, so it can be used to serve small resources. The keyword small is operative here: the data is limited to 64 KiB, which is plenty of space for storing interesting data, but not really big enough for storing images or any other sort of media.

As a social experiment, I think it's sort of fun: there's no authentication of any kind -- just come up with sufficiently unique keys to make it unlikely that anyone else will stumble upon them. Scouts honour that anything you put here is safe in our hands.

I'm curious as to what people think this might be useful for, and if you have any feature requests. I'd love to see people try to build something with it -- but go easy on us, since it's just an experiment. :)

25 comments

[ 3.6 ms ] story [ 66.1 ms ] thread
markchristian
Oh, I forgot to mention -- it's open source, too. http://github.com/shinyplasticbag/openkeyval
[dead] digispaghetti
I've already commented on the blog - but I'll say here as well - Looks like a great project, but seriously look at nodejs over PHP since JSON is native to nodejs, and you'll probably get more throughput on it.

Also, check out nStore and node-supermarket for existing native key/val stores already done in nodejs

aaronblohowiak
Clickable: http://openkeyval.org/

If you want to help people make webapps, promote the jsonp store to the top of the documentation and explanation of how it is used (including the fact that you aren't going to be able to store anywhere near 64kib with it -- more like ~2kb). While it is less neat, it seems to be the only viable solution for todays non-IE limitations on cross-site form posting.

bhiggins
hey sorry to burst your bubble but this is pointless.
markchristian
Nothing is pointless! Everything is an adventure. :)
heretoo
Exactly. I was trying to think how to build a chat program on this, and establish keys between two chatees for communicating, but it seems impossible to make a public channel/key for registering interest in chatting in the first place, without using an untrusted readwrite channel/key.

An append only store might solve it, with a key that is globally known. Some sort of broadcast channel. An append only, with limited size appends, and cleared at regular intervals.

markchristian
Good challenge! Here's a start:

Say you and I want to chat with each other. We both pick a random key to store our messages in. Let's call those key[markchristian] and key[heretoo]. Each of those has a read-only version -- readonly[markchristian] and readonly[heretoo]. I'll post the key readonly[markchristian] to "markchristian-to-heretoo". You post the key readonly[heretoo] to "heretoo-to-markchristian". Our chat clients can just poll those keys, looking for new messages -- whatever hand-wavy format we want to have.

Problems: 1. Requires us to agree to chat ahead of time and know each other's usernames. 2. Nothing to stop a malicious third-party from putting a different readonly key at the "heretoo-to-markchristian" pointer keys.

bhiggins
your enthusiasm is infectious in a bad way.
aaronblohowiak
This post is a great example of being "a hater". You aren't disagreeing about the application of a tool to a problem, you are dismissing all opportunity for anyone to enjoy or get value out of the item in question. Since the creators put effort in to creating this, it at least has value to them. So do the people that have upvoted it. So you're dismissing their opinion without offering any information, which is not very helpful.
bhiggins
fine. here's some information, then: you're promoting cargo cult thinking.
StavrosK
I share the GP's feeling, although not his short-sightedness. I would have made the same comment in a different way, namely:

I have no use for this and can't imagine where I would, so I can't give you any feedback. However, I'm sure that some people will find it useful and dream up new things to do with it, so good job!

EDIT: Actually! I think that this would be a great way to persist data in static, local html documents. You can probably create a full-blown web app with this in just HTML which someone can unzip and run in their browser.

markchristian
Your EDIT hit the nail on the head — that's exactly what I had in mind when I wrote this. I plan on writing some web apps that can run from anywhere as a pile of static files, using this as their backing store.
konad
I'm with you bhiggins
thepumpkin1979
Interesting, but maybe you can create some sort of namespaces so we don't override each other's keys.
markchristian
You can create your own namespaces by prefixing your keys. The whole idea of this is a flat, totally open namespace.
naz
Nice idea, super simple, and poses some interesting questions.

There could be security issues with this given you can serve arbitrary MIME types. XSS won't be a problem because there are no cookies to steal. Maybe you could make a script that self-propagates across the key-space using AJAX.

acme
Reminds me of OpenDHT, "a publicly accessible distributed hash table (DHT) service", which has since been taken down: http://www.opendht.org/ I do think being able to store information without too much hassle is pretty useful.
pudquick
... which in turn reminded me of the Vanish project http://vanish.cs.washington.edu/research.html which leveraged both OpenDHT and Vuze's (aka Azureus) DHT to store data (specifically: redundantly stored encryption keys, with the intent of letting them expire).
wazoox
Instead of storing the data in files, this looks like a perfect use case for TokyoCabinet or similar storage engines.
rlpb
Have you considered supporting the memcached protocol? It might help with takeup as client libraries already exist.

http://github.com/memcached/memcached/blob/master/doc/protoc...

markchristian
We have discussed it, and even considered just building an HTTP layer on top of it, but haven't done anything with it yet. As for supporting the memcached protocol, we considered that, too, and at this point, we're not targeting that use case. Personally, I've been planning on using this for entirely client-side JavaScript apps, where a memcache layer wouldn't be useful at all.

(That being said, the code is up on github, and patches are welcome. :)

stagas
I'm using a similar approach in a node.js module if you'd like to try it. http://github.com/stagas/chaos stores the data in files using the sha1 hash key like you do, you could easily implement an http api layer on top of it. I get about 2k writes/s and 4k reads/s on a regular hd, and 40k-60k writes/reads per second on a ssd.