Interestingly enough, Alan Shreve gave a talk at RedisConf a couple of years back detailing something very similar used on the twilio stack to rate limit draining a queue: https://inconshreveable.com/talks.html
Reading the title, I was expecting to see a somewhat shady article on using multiple servers to dodge API rate limits.
Instead, they've got a pretty snazzy writeup of how to effectively keep your distributed processes working together properly. I'm curious why they didn't pass a token rather than the actual Lua over the wire via Redis, but it certainly seems to work for them.
So they send code to Redis to make sure that they won't increment too high? I know Redis already has a ton of functions prepackaged but this seems like it would be a good one: INCRIF.
I may be missing something but I think this could be done more cleanly with an atomic counter or semaphore [1] in Redis.
1. Have a task that releases 1 resource to the semaphore
every (1.0 / cps) seconds.
2. Workers wait to acquire a resource from the semaphore
before making a call.
Due to the rate of release being fixed at 1/cps no worker can ever exceed the calls-per-second limit. Much simpler than sending Lua over the wire.
The only downside I see to this approach in the context of this post is that then you'd essentially need to dedicated daemon sending releases to the semaphore for #1.
I suppose you could use a long running celery task for that purpose. You wouldn't want to fire of (1.0 / cps) celery tasks just for releasing semaphores unless the value of cps was very small -- as it would congest your queues with a bunch of small cleanup tasks. Which aren't guaranteed to run precisely that often.
5 comments
[ 3.1 ms ] story [ 24.6 ms ] threadInstead, they've got a pretty snazzy writeup of how to effectively keep your distributed processes working together properly. I'm curious why they didn't pass a token rather than the actual Lua over the wire via Redis, but it certainly seems to work for them.
[1] https://github.com/dv/redis-semaphore
I suppose you could use a long running celery task for that purpose. You wouldn't want to fire of (1.0 / cps) celery tasks just for releasing semaphores unless the value of cps was very small -- as it would congest your queues with a bunch of small cleanup tasks. Which aren't guaranteed to run precisely that often.