Related: Redis 1.1 (currently in beta but ready for some test) implements a sorted type supporting range queries. It's pretty useful when the need is to take million of items sorted with very fast top-N items operations or items with score between min and max, especially in presence of continuous updates to the items scores.
If you are interested check http://code.google.com/p/redis/wiki/CommandReference (All the commands prefixed with "Z" are about the Sorted Set data type). Btw both insert / remove / and update-score operations are O(Log(N)). Top-N items is O(N) since the list of items is already sorted.
Redis 1.1 will be released as stable in one or two months at max.
For data structures and algorithms geeks: the sorted data type of Redis is implemented with a skip list, hacked to actually build a doubly-liked list, with the backward pointers being only at level 0 (we have reverse range operations, ZREVRANGE). Every element is also taken inside an hash table, so that it's possible to update the score of an element already inside the sorted set in O(Log(N)) time because we can get the old score in O(1) from the hash table, and then find the element in the skip list.
Even when the scores are not distributed but there are large clusters of elements with the same score, the update is still O(Log(N)) thanks to some interesting trick.
Redis is really cool, I can think of lots of uses for fast and lightweight and there's at least one project where it would have saved me a lot of effort if it had existed then!
Watch out for the Python library though! It doesn't even try to sanitize key names so malformed names cause all sorts of problems (including executing arbitrary commands). Plus unicode strings with non-ascii chars cause it to blow up.
All very easy to fix of course, as soon as I get a chance I'll submit a patch if it hasn't already been done by someone else by then.
Hello almost, I don't know Python so I'm not sure about this problems, btw starting from Redis 1.1 the client libraries will no longer be included in the main project (now they are in the tarball of Redis), and I'll try to stimulate much more a Bazar development style for client libraries.
Btw the Ruby client lib is solid, like it appears to be the Java one. Still there are client libs that absolutely need to be improved.
Currently the Ruby one is as far as I know the only one supporting consistent hashing but probably this problem will be fixed with the introduction of a new daemon 'redis-cluster' that will work as a proxy taking care to deal with the hash ring in a transparent way.
Check the changed doctests for examples of why it was a bad idea not to sanitize key names and other params. Ignore the changed doctests involving Decimals, I had to change those to get them to pass on my machine but they're not related to me fixes.
Redis is particularly useful for implementing simple message queueing systems where the full functionality or power of something like RabbitMQ is not required.
For example, when parsing incoming data into a database, push incoming data onto one end of a Redis list, and have a seperate worker process popping them off the end and into CouchDB, MySQL or whatever.
The fact that Redis list operations are atomic mean that multiple worker processes (possibly on seperate physical boxes) can process jobs simultaneously.
Result: the internet is entirely decoupled from both the hard drive and the database, and extremely high performance is possible.
Is Redis stable enough to replace using a database as persistent storage. There seems to be some indication that it is, but then there are paragraphs in the documentation that makes it seem like it's not. Particularly if your database size exceeds the machine's memory then you could lose data.
Is one of the goals of the project to replace the need to use a database? Or will it always be more suitable cache or worker queue replacement?
Hello, the documentation should be updated indeed. Now Redis supports the maxmemory directive that will start to reply with an error on write commands when the database is using more than the specified bytes of RAM.
There was a post earlier in the week about why Redis was "awesome" and a "lifestyle" and I posted a kind of snarky comment about why posts like that scare me away from Redis.
After this post, I had an hourlong discussion with my lead dev on swapping out memcached for Redis so we could use it for our job scheduler. This post makes an excellent, compelling case for Redis.
Just an interesting example of how two different evangelistic approaches can play out. I don't know if I'm representative of "normal people", though. (Don't say it.)
It was that snarky comment on my first post on "awesome" and "lifestyle" (although I thought I'd make pretty clear that I don't consider Redis a lifestyle) that made me write this second one. I'm glad it helped clarify some things.
Indeed. I went to a CouchDB presentation at a local conference, and afterward I asked about some transaction-related details. The presenter didn't understand, but kept repeating that the technology behind it was "way sexy". It was useless and off-putting.
15 comments
[ 5.5 ms ] story [ 49.1 ms ] threadIf you are interested check http://code.google.com/p/redis/wiki/CommandReference (All the commands prefixed with "Z" are about the Sorted Set data type). Btw both insert / remove / and update-score operations are O(Log(N)). Top-N items is O(N) since the list of items is already sorted.
Redis 1.1 will be released as stable in one or two months at max.
For data structures and algorithms geeks: the sorted data type of Redis is implemented with a skip list, hacked to actually build a doubly-liked list, with the backward pointers being only at level 0 (we have reverse range operations, ZREVRANGE). Every element is also taken inside an hash table, so that it's possible to update the score of an element already inside the sorted set in O(Log(N)) time because we can get the old score in O(1) from the hash table, and then find the element in the skip list.
Even when the scores are not distributed but there are large clusters of elements with the same score, the update is still O(Log(N)) thanks to some interesting trick.
Watch out for the Python library though! It doesn't even try to sanitize key names so malformed names cause all sorts of problems (including executing arbitrary commands). Plus unicode strings with non-ascii chars cause it to blow up.
All very easy to fix of course, as soon as I get a chance I'll submit a patch if it hasn't already been done by someone else by then.
Btw the Ruby client lib is solid, like it appears to be the Java one. Still there are client libs that absolutely need to be improved.
Currently the Ruby one is as far as I know the only one supporting consistent hashing but probably this problem will be fixed with the introduction of a new daemon 'redis-cluster' that will work as a proxy taking care to deal with the hash ring in a transparent way.
Check the changed doctests for examples of why it was a bad idea not to sanitize key names and other params. Ignore the changed doctests involving Decimals, I had to change those to get them to pass on my machine but they're not related to me fixes.
For example, when parsing incoming data into a database, push incoming data onto one end of a Redis list, and have a seperate worker process popping them off the end and into CouchDB, MySQL or whatever.
The fact that Redis list operations are atomic mean that multiple worker processes (possibly on seperate physical boxes) can process jobs simultaneously.
Result: the internet is entirely decoupled from both the hard drive and the database, and extremely high performance is possible.
Twitter Streaming API, anybody?..
Is one of the goals of the project to replace the need to use a database? Or will it always be more suitable cache or worker queue replacement?
After this post, I had an hourlong discussion with my lead dev on swapping out memcached for Redis so we could use it for our job scheduler. This post makes an excellent, compelling case for Redis.
Just an interesting example of how two different evangelistic approaches can play out. I don't know if I'm representative of "normal people", though. (Don't say it.)