I think the entire point of this post is just to share a war story and help others understand some Redis problems and debugging steps. It isn't some deeper optimization guide or anything. This exact problem is something I have caused and encountered multiple times in my career now. On my first day at Instacart the entire site was down in an extended outage due to a hot key issue exactly like this. I think stories like this are worth sharing with the community even if it seems trivial to yourself.
This is team of 2 engineers that's building + shipping faster than most teams I've seen while having insane number DAUs. Moving fast (and patching problems even faster) is the name of the game here and these guys delivered.
8 comments
[ 0.21 ms ] story [ 29.9 ms ] threadThey patched out a gross inefficiency that they created
That said, the diagnostics exhibited here are the exact things I'd do, as a person who visits Redis once maybe every two years.
They're not exactly groundbreaking, and I was hoping to find revelations here; not basics.
Looking at what the cluster as a whole is doing, and a specific node, and trying to move it. Amazing.
You'd hope that code reviews would catch this kind of thing, especially on a small team.