HN Explorer

Using prime numbers to avoid cron task collision at sametime

2 points by akshayB ↗ HN
I would like to get input from you all. We have legacy system (bit of a pain to maintain) where multiple corn jobs runs (8 to be precise) and engineers who created the system put everything to run at 5 minutes interval. Computationally task is heavy and often results into a crash or running out of memory but business owners are fine if things are delayed by like an 1 hour or so.

My idea was to run on prime intervals instead of every 5 minutes. Couple of them have depend on output from others tasks but that should not be an issue.

Job 1 - Every 5 minutes ( computational less intense )

Job 2 - Every 7 minutes

Job 3 - Every 11 minutes

Job 4 - Every 13 minutes

Job 5 - Every 17 minutes ( computational more intense )

.... and so on

This should put a significant dent in avoiding them running at same time and leaving some room for each task to finish itself. I was just wondering if there is any other alternative that I can explore here.

Thanks for any suggestions in advance.

6 comments

[ 4.8 ms ] story [ 24.7 ms ] thread
CarolineW
If the owners don't care then just have

  */16 * * * * bin/RunJob_1
  2-59/16 * * * * bin/RunJob_2
  4-59/16 * * * * bin/RunJob_3
  6-59/16 * * * * bin/RunJob_4
  8-59/16 * * * * bin/RunJob_5
  10-59/16 * * * * bin/RunJob_6
  12-59/16 * * * * bin/RunJob_7
  14-59/16 * * * * bin/RunJob_8
If that doesn't work for you then tell us why. As it is your problem is underspecified.

Also, you say some are more compute intensive - have you actually measured the load and time-to-finish?

Prime numbers and similar techniques can help, it just depends if you actually just want to kludge something together, or do it properly.

ultrablue
Is your goal to avoid all collisions?

If so your current scheme won't prevent that. For example, Job 1 and Job 2 will both run at 0035, 0405, 2025, etc. (assuming that both jobs start at 0000).

CarolineW
That depends - cron time steps always restart from 0 in the hour, so what you say is valid, but the most obvious way of implementing what I think is being suggested doesn't have exactly that problem.
akshayB
yeah I am ultimately trying to avoid collision and because other route for code refactor is not an option
pwg
> I was just wondering if there is any other alternative that I can explore here.

Investigate task spooler: http://vicerveza.homeunix.net/~viric/soft/ts/

This will let you queue up the tasks, and each will run as soon as the last has finished, but you won't get a "thundering herd" where they all are trying to run at once.