I'd add an additional safeguard. When retrying failed jobs, make sure to have exponential backoff. Numerous queueing systems have this feature built in.
Good point! Delayed Job actually has an exponential backoff build in. It'll run 25 times, with an increasingly large time between attempts before it gives up.
The issue comes in when it doesn't know a job actually failed, as was the issue in this case. Since the process was `kill -9`'d delayed job wasn't aware the job failed and attempted to run it again as though it was the first attempt.
2 comments
[ 3.1 ms ] story [ 16.9 ms ] threadThe issue comes in when it doesn't know a job actually failed, as was the issue in this case. Since the process was `kill -9`'d delayed job wasn't aware the job failed and attempted to run it again as though it was the first attempt.