I can see one use case for this family of tools: if the same script could generate either crontab files or launchd agents, so it could generate scheduling files for MacOSX, where crontab is discouraged.
Celery beat is pretty much exactly the same thing as cron except it's another service to configure and worry about keeping up and running. And it uses 35 MB of RAM instead of...well, I guess cron uses a few kilobytes.
I can't see the point of it at all.
When I want to kick off celery tasks I just have a cron job execute a django shell command.
I don't get it. What's so hard about cron jobs that you have to write a wrapper for it? It is easy to understand if you just read the crontab file comment at the top.
There is just two commands:
crontab -l
for listing
crontab -e
for editing. What else do you need?
With this, instead of learning crontab simple syntax, you need to:
1. You can validate cron jobs and programmatically change or reject some of them. Or maybe just offer a small set of services as a cron job.
2. Once you have a security model you can expose this as a web service and offer cron jobs over the web.
3. You could extend this API to use different implementations of cron. Instead of calling the actual cron you could implement cron in Python or maybe use a cron web service.
1. crontab -e command automatically recognize if you have a syntax problem in the newly edited file:
$ crontab -e
crontab: installing new crontab
"/tmp/crontab.QhkOLb/crontab":25: bad minute
errors in crontab file, can't install.
Do you want to retry the same edit? (y/n)
I wrote something similar a while back, but it only requires an hourly cronjob to run a bunch of Python scripts in an arbitrary file structure: https://github.com/nlindblad/ocarina
I use it extensively myself for scraping various sites/APIs.
I'd typically deal with this by exporting the functions to be ran as entry points in the the setup.py file, and then use whatever configuration management tool I'm using (Ansible, Puppet, whatever) to manage the crontab file in /etc/cron.d. Ansible's cron module makes this especially easy.
I see the value in this: it provides a nice programmable interface to cron. Unlike what you might think at first, cron has some really interesting idiosyncrasies that you might not appreciate at first. For example, if your filename in /etc/cron.d/ has a "." character in it, it will be silently skipped [1]. Also, files in /etc/cron.d/ will not inherit environmental variables from /etc/crontab. Consider how many hours have been wasted "debugging" these issues.
That said, cron is easy to abuse as well. The example of providing exponential backoff is kind of silly: there are better tools for the job, including Celery. Cron also does not have seconds precision which is a bummer for lots of short-lived jobs. Lastly, cron only runs jobs at the specified time. If you adjust the system clock, or shut the system down, your tasks might be skipped. anacron [2] might be a better solution in some ways. Lastly, all implementations of cron that I am aware of don't have a great ability to make sure tasks actually run and finish, don't have retries built-in, and have a horrible interface for reporting errors (any output of the script is emailed to root@localhost from root@localhost).
Based on the above, you might think I am saying to never use cron. That's not at all true. Cron is extremely useful as a systems tool. If you package your software in the distro's package format (you should at least give this a try to see how the system is put together), cron is pretty much your only safe choice. Also, as long as you follow the rules cron sets up, and know of its potential issues, cron will be the workhorse that gets things done. Tasks like log rotation, periodic sanity checks, etc. are great for this. If you are writing a web application and are already using Python though, look at more sophisticated options such as Celery before using cron directly.
18 comments
[ 4.2 ms ] story [ 51.8 ms ] threadI don't see any use case for Plan, it's just complicating an easy task.
I can't see the point of it at all.
When I want to kick off celery tasks I just have a cron job execute a django shell command.
A full megabyte on my machine, actually! Still a long way from 35MBs, though.
With this, instead of learning crontab simple syntax, you need to:
- install this package
- learn its API
- run it and still edit the crontab file :D
seems like not worth it.
If it's because users can only have a single crontab, you can always use system crontabs instead or "cat" them together.
For instance, server side exponential backoff.
1. You can validate cron jobs and programmatically change or reject some of them. Or maybe just offer a small set of services as a cron job.
2. Once you have a security model you can expose this as a web service and offer cron jobs over the web.
3. You could extend this API to use different implementations of cron. Instead of calling the actual cron you could implement cron in Python or maybe use a cron web service.
I use it extensively myself for scraping various sites/APIs.
That said, cron is easy to abuse as well. The example of providing exponential backoff is kind of silly: there are better tools for the job, including Celery. Cron also does not have seconds precision which is a bummer for lots of short-lived jobs. Lastly, cron only runs jobs at the specified time. If you adjust the system clock, or shut the system down, your tasks might be skipped. anacron [2] might be a better solution in some ways. Lastly, all implementations of cron that I am aware of don't have a great ability to make sure tasks actually run and finish, don't have retries built-in, and have a horrible interface for reporting errors (any output of the script is emailed to root@localhost from root@localhost).
Based on the above, you might think I am saying to never use cron. That's not at all true. Cron is extremely useful as a systems tool. If you package your software in the distro's package format (you should at least give this a try to see how the system is put together), cron is pretty much your only safe choice. Also, as long as you follow the rules cron sets up, and know of its potential issues, cron will be the workhorse that gets things done. Tasks like log rotation, periodic sanity checks, etc. are great for this. If you are writing a web application and are already using Python though, look at more sophisticated options such as Celery before using cron directly.
[1] http://manpages.ubuntu.com/manpages/maverick/en/man8/cron.8.... and https://bugs.launchpad.net/ubuntu/+source/cron/+bug/706565
[2] http://en.wikipedia.org/wiki/Anacron