HN Explorer

Redis vs. KeyDB vs. Dragonfly vs. Skytable

130 points by johnwoods ↗ HN
It seems that several projects claim that they have built the "world's fastest key/value store" or sometimes the phrase used is even more outrageous. The following projects are in question:

- Redis: https://github.com/redis/redis

- KeyDB: https://github.com/snapchat/keydb

- Dragonfly: https://github.com/dragonflydb/dragonfly

- Skytable: https://github.com/skytable/skytable

And, these are my benchmark results (30 threads, 1 client per thread with GET/SET. See below for setup):

1. Redis: 112,100 / 99,892

2. KeyDB: 288,931 / 282,997

3. Dragonfly: 408,322 / 392,446

4. Skytable: 619,992 / 676,091

# Notes

1. Redis: I'll start with Redis which I'd like to call the "original" key/value store (after memcached) because it is the oldest and most widely used of all. Being a long-time follower of Redis, I do know it's single-threaded (and uses io-threads since 6.0) and hence it achieves lesser throughput than the other stores listed above which are multi-threaded, at least to some extent. The best parts about Redis: it's the most feature complete of all the systems here, and is the oldest. (not saying old necessarily means better).

2. KeyDB: The second is KeyDB. IIRC, I saw it in a blog post which said that it is a "multithreaded fork of Redis that is 5X faster"[1]. I really liked the idea because I was previously running several instances of Redis on the same node and proxying them like a "single-node cluster." Why? To increase CPU utilization. A single KeyDB instance could replace the unwanted proxying funkiness, so I ditched Redis for KeyDB. Has been a fine experience so far, except for some occasional crashes.

3. Dragonfly: Just found it on HN and seems to be relatively new. Dragonfly claims that it is 25X faster (that I couldn't reproduce) than Redis[2] and has the slogan "Probably, the fastest in-memory store in the universe!". Doesn't support all the Redis commands yet, but I find it interesting mainly because of performance. Also, it's good to know why it is faster because it clearly outlines the underlying architecture[2]. The other three stores don't say much about it. Will be following the project.

4. Skytable: Found it while looking for projects written in Rust. Claims to be "insanely fast." Skytable's "experimental benchmarks" claim that it is something around 10X faster than Redis and some 2X-3X faster than KeyDB[3]. I hadn't heard of Skytable and it doesn't seem to be as widely used (unless I'm missing something?). I find it interesting because of the planned features[4] and performance. Only Skytable natively runs on Windows out of the four. Will be following the project.

5. My thoughts: Redis needs no introduction and is arguably super stable for use on production systems (using widely in our systems). KeyDB seems to be "stable enough" and it seems to be good for prod since Snapchat uses it already[5] (and so do we!). I found no Dragonfly v/s Skytable benchmarks. The best part about Redis, KeyDB and Skytable is that they don't make any "crazy assumptions" about the system they run on. What do I mean?

Dragonfly expects you to have the latest hardware[6] and the latest kernel[7] on all your servers. I find this outrageous because not all servers run 5.10 and a majority of them are still using long-running branches on 4.x. Also, I don't expect them to have the latest processors, either. I'd argue if the other three stores started assuming the latest features, they'...

55 comments

[ 2.3 ms ] story [ 128 ms ] thread
w-ll
Have YOU tried benchmarking any of them locally or in the cloud?
johnwoods
updated the post
w-ll
I use Redis because of some of the extra features for sets and queues and such.

If I just wanted pure kv store i'd still go memcache. local install for stuff that can expire often (html template caches) and reach out to a network store for user sessions and the like.

all my kv caches are designed to be rebuilt and not care. if you need ACID kv go with a proper db.

viraptor
> Dragonfly expects you to have the latest hardware[6] and the latest kernel[7] on all your servers. I find this outrageous because not all servers run 5.10 and a majority of them are still using long-running branches on 4.x.

I don't see anything wrong with that. Either your kv store is performing OK or can be scaled using the usual methods and it's not holding you back... or you actually critically depend on the fastest kv performance you can get. In the second case, you'll be likely happy to dedicate some time to standing up a couple of machines with updated kernels.

tyingq
As far as I can tell, "latest kernel" really means "has io_uring"...which is core to how Dragonfly works, so it seems reasonable to me.
websap
If you're running a really old version of Linux, you're probably giving up on many-many performance optimizations.

> I find this outrageous because not all servers run 5.10 and a majority of them are still using long-running branches on 4.x

I don't think it's outrageous to ask people to upgrade 5.10. This was released over 1.5 year ago and there's no reason why your corporate overload shouldn't consider migrating to this, or at least have someone working on qualifying their workloads.

forty
One important thing I would look at is "is there independent managed hosting options", in case at some point you have better things to do than managing database and having to care if it's the right hardware and kernel version. I know at least redis shines on this, with managed offering on all major cloud provider.
cientifico
Why do you consider single threaded as a reason to have less throughput?

Are you looking for throughput or speed?

Also, what are your needs?

codetiger
He has mentioned that to gain better CPU usage he runs multiple redis on the same machine. Probably the only issue is unused CPU
bradknowles
Sorry, you say you ran the benchmarks, but where in this post do you link to your results?
johnwoods
updated the post with the overall result for now
ksec
>Redis, which I'd like to call the "original" key/value store

My old friend Memcached just came by and want to say Hello. He is still alive and kicking. Netflix has been treating him well.

lapser
Wow! I was under the very wrong impression that memcached hadn't been updated in a long time. It has been having regular releases for longer than I thought. Great to hear!
ksec
Somewhere along the ling, lots of ( well deserved ) hype went to Redis and Postgres, and people thought Memcached and MySQL died.

I have been wondering for a long time if we will ever see Memcached 2.0 and MySQL 9.0

johnwoods
> My old friend Memcached just came by and want to say Hello. He is still alive and kicking. Netflix has been treating him well.

Forgot my dear buddy. Added it back.

[deleted]
(comment deleted)
zeckalpha
It has cache in the name so people are less likely to use it inappropriately.
blagie
My major use-case for a KVS is *reliable, scalable* storage. RDBMS has an upper bound on scaleability. Removing the relational component removes a lot of power and flexibility, but improves performance and scalability.

That's often a good choice!

A KVS can be arbitrarily horizontally scaleable. That might not sound necessary for a small project, but even there:

1. You can deal with big data (e.g. keystroke-level event data, etc.)

2. You don't have to worry about scaleability. It's one less thing to worry about.

3. You're set up for success if you take the unicorn route.

That's not memcached. I like memcached, but it's not the same as a KVS.

dmitriid
Redis and sqlite are two projects I can't really believe exist. They both are absolutely magical.
Cwizard
Could you explain why?
dmitriid
Both are small projects by small teams (though I don't know how small Redis team is now). They have insane performance characteristics with rather small footprints and a wealth of functionality.
blagie
My experience is that small, dedicated teams of high-performing individuals are the only way to build tools like these. The level of architectural and design coherence starts to decrease at 5 people, and falls like a rock beyond 10-20.

The key problem is building a firewall so that such people can focus. The *organizational* problems there are hard. If you have a firewall around five people, you can't tell if:

- They're sitting at home drinking beer and watching cat videos

- Are developing the world's fastest database

Which leads to meetings, presentations, funding proposals, and so on. Which, in turn, leads to the need for large teams.

cmacleod4
.. and both happen to have strong links to Tcl :-)
judofyr
> I was NOT able to reproduce the 25X speed that Dragonfly claims

The Dragonfly benchmark which showed 25x faster was run on c6gn.16xlarge which has 64 vCPU. The m5.4xlarge you used "only" has 16 vCPU. Since Dragonfly is achieving its performance by using multiple threads it's not weird that you weren't able to replicate its performance gain.

In my opinion, when it comes to these types of multi-threaded benchmarks it's much better to separate the "baseline, one-process performance" from "how it scales with number of processes". E.g. if you first pin Dragonfly to only run on a single core you can find the baseline performance compared to Redis, and then you can run different benchmarks of Dragonfly with increasing number of cores to see how it scales. This should give you much better understanding of how the system behaves. (The release of Dragonfly actually prompted me to write a blog post on this: https://t.holmium.no/dia/apples-vs-oranges/)

ghusbands
If you start with "baseline, one-process performance", you're likely to mislead yourself, since no modern server has fewer than eight, and a single-thread benchmark can omit important locks. A modern baseline is probably two to four cores, even if you're running multiple services on one machine.
vlovich123
I think you misunderstand. You start with 1 then test 2, 4, 8… up to the largest core count you can. This shows you where the multi threading arch breaks down.
johnwoods
agree with your article entirely. the dragonfly benchmark seems far too artificial to be true and used in practice (given the expensive hardware). my tests were on a relatively cheaper box, and Skytable topped all the other systems.
spaintech
This article interests me as it has been a bucket list todo to get a good sense of some of the key/value offerings, thank you for posting them.

Quick ask: I don’t see “some” of the other offering out there like MemCached… what was the criteria used to select these? I don’t see any source of how the test where run, specs of the systems, how the DB where set up, etc. Would be very valuable to have in order to attempt to re-validate these test on our own platform. I also came back and saw some of your updates to the outcomes, While the Get/SET numbers are provided, I don’t find a reference to time, resource utilization, or time which would IMO make the data more valuable in understating how the numbers where achieved. * I might be a factor of the benchmarks themselves, but would add value IMO to your fantastic post.

fmajid
KeyDB also has another key advantage over Redis: if your data set exceeds RAM, it will spill to disk (SSD) and maintain good performance.
FridgeSeal
> Dragonfly expects you to have the latest hardware[6] and the latest kernel[7] on all your servers. I find this outrageous because not all servers run 5.10

I don’t understand why this is “outrageous”? There’s performance features and better hardware utilisation in newer kernels? If you want the performance offered by the applications built on top of these, then arguably you should be prepared to run the hardware and software required to achieve this.

johnwoods
not if I can get equivalent (if not better) performance on my existing hardware
FridgeSeal
From your own numbers, Dragonfly get ~3.6-4 times better perf than Redis, and better than keydb (which is “just” multhreaded redis-simplifying). That indicates pretty clearly to me that we’re not getting same-or even close performance-on existing hardware; and ultimately, if performance is that important, shouldn’t we be giving our applications every advantage they can get?
TheChaplain
I'm trying to get my workplace to accept Redis for simple caching but they're not open to the idea. Instead they're all in Hazelcast but real stingy about who gets to use it and for how much memory.

I just want to cache some objects and avoid hammer the DB on every request :´(

ghusbands
You could probably find a local caching library to do the same thing in-process (via mmap) and partially reduce workload that way. Hopefully, using a local library needs less managerial involvement than installing/using a service.
romange
Hi, I am the author of Dragonfly. Following this post, I performed the loadtest on instance m5.4xlarge, recorded myself and posted the video here: http://assets.dragonflydb.io/videos/video2609676488.mp4

the command I used to load was: "memtier_benchmark --ratio 1:0 -t 28 -c 20 -n 100000 --hide-histogram --distinct-client-seed -d 256"

I did the recording to clear out any doubts whether my benchmark results are real.

johnwoods
please note that I ran the bench with 30 threads with 1 connection per thread. you're not doing the same but doing 28 threads with 20 connections on each thread, effectively using 560 connections whereas I only used 30 connections. rerun the same with -t 30 and -c 1
tmikaeld
I think the point he/she is trying to make is the 25x claim is real
johnwoods
if I'm doing my math right: redis is doing 99892 sets per sec, so 25X should be 2,497,300. the video however shows 1,023,030 which is more like 10X and NOT 25X
tmikaeld
Agreed, emphasis on the word _trying_
didip
Anyone knows where can I find a good helm chart for KeyDB?
benedetto73
Redis can do 1M or even 2M requests per second on a decent box. What payload are you using? Can we see the script you are using to generate the traffic and measure the performance?
jdsully
KeyDB author here. Lack of stable kernel support was a major reason we didn't do io_uring yet. The goal for KeyDB is good performance out of the box not in specific benchmarks on specific hardware/kernel versions. As io_uring becomes mainstream we will start to support it. In 6.2.3 we have async commands which permit reads to happen outside of the global lock. In some cases it can take extra time for writes from other clients to be visible which changes the consistency model so it is off by default but dramatically improves performance when enabled.

Doing a fresh rewrite would have enabled a nicer threading model, but it would make it harder to maintain compatibility. Plus at the time it was more to suggest to Redis multithreading was both desirable and possible which were disputed back then.