29 comments

[ 2.5 ms ] story [ 68.3 ms ] thread
I don't really understand. What could I do with this that I couldn't do with cron? Or is it that somebody who can't use cron could use this? Or that this is more reliable than cron?
(comment deleted)
While cron is a nice tool, people should not be using cron for any production grade automation.

Tools have been created to that orchestrate automation with more powerful capabilities, and Airflow has become a defacto standard in the industry.

Prefect is an open automation tool, comparable in a lot of ways to Airflow. The author of the post is an early committer to Airflow.

>While cron is a nice tool, people should not be using cron for any production grade automation.

?????????????????????????

Because many decades of stability based upon simplicity are not good enough for today's world.
I don't really get into this argument anymore - I'm convinced opinions like this are held tighter than religion or politics.

Here are a few thoughts, written by others, that I would probably bring up if I wanted to get into this argument:

https://medium.com/eshares-blog/why-you-shouldnt-use-cron-jo...

https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextd...

https://medium.com/videoamp/what-we-learned-migrating-off-cr...

https://medium.com/@rbahaguejr/airflow-a-beautiful-cron-alte...

As a point of reference, I run a company that brokers terabytes of mission critical data each month for our customers. I've advised dozens of companies on how to properly ensure scheduled jobs are maintainable, instrumented, logged, and reliable. The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense.

(comment deleted)
Your view is exceedingly narrow and uninformed if you really believe running Cron in production to be a blanket "terminable offense". A tool not being sufficient for your use case does not make that tool worthless or poor. You should reflect a bit before making statements like that if you actually run a business.
I’ll only touch on the first link because it's late. Haven't looked at the others.

> Smallest resolution is 1 minute — If a task needs to run every 30 seconds, you can’t do it with cron.

This is the only one I can see that could be an issue. But engineering wise, I don't think this is the best solution (OP's). You still have overhead if you're launching from an external tool. I'd probably run this inside the application (like they do here).

> Error handling — If a job fails, what should happen? Solutions have been built to solve this single problem. Developers love adding more band-aids rather than admitting there is a better way. Deja Vu?

If it fails to launch or what was supposed to run failed? Going to assume whatever was supposed to run failed since any application can fail/crash. Even celery beat which is what they propose here.

You handle the condition. The same thing you'd do anywhere,

    true && echo “True” || echo “False”
    false && echo “True” || echo “False”

> Logging — Crons don’t log, unless you tell them too. It is rare to see people log output of their cron jobs.

Cron by default mails stdout and stderr. This is a log. Regarding the second point, I'm assuming it means redirects. If so, that's an issue of misconfiguration and not understanding your tool. Same with any tool/program.

> Working with cron pulls you out of the application — cron is a system level process. Not an application process.

This suggests another process to run jobs (`celery -A proj beat`). It’s another dependency to manage and requires you to launch it. So it still handles an external process.

> It’s challenging to give application developers access to anything at a system level. Developers shouldn’t care where their application runs… A good example of this is timezones. If a system person changes the timezone of a server, the cron may run at a different time than expected. The less the app developers have to worry about what they run on, the better.

So the author is saying this applications doesn’t know about time zones? Celery Beat does know about them. It works on UTC by default but it can be configured.

Regarding giving developers access to system settings... why? This article says they devs can't log cron correctly or deal with timezones, allowing all devs to be able to manipulate this is crazy. If the server runs more than one thing, the mess would be horrible.

Timezone misconfiguration on the server, or any kind of misconfiguration, could be an issue. But so is an applications being misconfigured and running on a wrong timezone. If teams can do things like changing timezones without coordinating and talking, that's an internal issue.

> The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense.

Except for the 30s resolution, I haven't seen a good reason on this article that doesn't boil down to misconfiguration, not knowing how something works or communication.

30s resolution can be done easy enough,

    #!/usr/bin/env bash
    while [ true ]; do
      sleep 30;
      #run whatever you need. 
      # Append & if you don't need to wait for it to finish 
      /path/to/program
    done
And then just create a service for it on system.d with `Restart=always`.
Addressing one of those articles, "What we learned migrating off Cron to Airflow (https://medium.com/videoamp/what-we-learned-migrating-off-cr...):

...changes to the crontab file were not easily traceable...

Don't edit the crontab. Use cron.d entries, atomic files dedicated to specific packages, services, and/or tasks. In a multi-host environment, manage those under your configuration management tool.

Mind: a major gripe I've had with Chef in the past was that it failed to use individual cron.d files and instead edited the crontab file directly under its cron management interface. That is a critical design error.

Cron keeps a log of job outputs on the server where the jobs were run

That'a fault of your logging system, as all system services log locally by default. Set up a log server if you need centralised logging.

...bash profile settings have been stripped from the Cron environment...

Sounds like someone unfamiliar with cron, or sane script development and testing procedures. If you need environments set, set or source them explicitly (I recommend the latter, FWIW).

These are PEBKAC, not cron, errors.

cron stable? I remember having to add a cronjob to touch a file every 5 minutes and having Nagios monitor that because occasionally it would just stop processing new jobs and have to be restarted

That may have been a Centos5/6 specific thing and it was probably caused by too frequent -HUP ing it, but it didn't feel like the decades made it that stable..

My understanding of why using cron for background tasks with web apps isn't great in production is keeping track of things when you have distributed servers. I think cron is pretty limited in scope to whatever machine it is running on.
Look, don't get me wrong, cron is a super useful tool for doing system-level tasks on a schedule but "grep a timestamp and run a program if it matches" is practically the definition of "doesn't scale."

Unless you code it yourself in your program cron doesn't...

- understand dependencies

- run jobs across multiple hosts

- handle errors (e.g. if err run this)

- do any branching whatsoever

- handle timespecs with more granularity than 1m

- do any reporting (mail counts-ish but if you're telling me you have an email based automation flow I won't believe you)

- do any metrics collection

- run pools of jobs

- understand priorities

- allow you to queue jobs

The value prop of things like airflow is that they factor all this logic out of your applications. At the point where you need these things "run a thing at a specific time" is the most trivial part.

I think you should write their homepage as your comment is much more informative.
> - understand priorities

> - allow you to queue jobs

technically this can be done oldschool with cron+at/batch.

Then it's the wrong comparison, because cron simply schedules other commands, it's not a programming language.
Yes, chron is just a very simple, single-node, low-time-resolution, system-level scheduler. While it may be good at that, that is very often (always would be an exaggeration, probably) not a great tool to be used (at least as an exposed component) in automation of modern production systems.
Cron is a scheduler. If you need other features, your cron script(s) themselves can check those.

Addressing your specific items:

Dependencies. Depending on which of these you're referencing, the package manager (which installs a given cron script), configuration manager, or script itself, can and should test and check these.

Multiple hosts. There are numerous multi-host utilities, ranging from SSH extensions / utilities to full multi-host configuration tools (Ansible, Salt, Chef, cfengine, Puppet, etc.) At heart, this isn't cron's problem. It's a schedular, not an orchestrator.

Error handling. Your script's issue, not crons. Cron is a schedular, not an error handler.

Branching. Not sure what you're referencing, but again, the task of the script, generally.

Sub-minute granularity. Cron is a schedular, not a daemon substitute. If you need a daemon operating at sub-minute granularity, write a daemon.

Reporting. This is the job of your logging facility, generally.

Metrics. This is the job of your accounting facility: systat, syslog, etc., etc.

Pools of jobs. This is the job of your script(s), generally. If you have a set or sequence of tasks, trigger the master with cron, and have it manage the set. Again, cron is a scheduler, not an orchestrator.

Priorities. Nice.

Queue jobs. You want 'at' or 'batch', not cron. Cron is a scheuler, not a job queue.

Pretty sure a lot of production systems use cron
this appears to handle interdependencies / multi machine / in-application hooks

you could create some cron framework + glue code that accomplishes the same thing, but that would be alot of code.

What’s the context for this? Seems like it’s a marketing piece to get people to sign up for their fake scarcity early access program.

Very little interesting in the article.

You can't possibly have read the article.
Your comment is in violation of HN guidelines.

All I saw is a marketing piece designed to tell me about the product line of a company I’d never heard of. No context as to things I could relate to, mostly just information and excitement about this prefect platform.

Outside of prefects products and features, what is the information you took away from this article that made it worth your time?

I read it and came to a similar conclusion as the parent poster.
How does this compare to Airflow? It's unclear from the website - can you create DAGs (task sequences)?
Yeah, this introduction/launch needs a lot of work. I've read over the first paragraph or two and all I see is buzzwords: "turn-key", "orchestration", "cloud", etc...

If you can't explain your value proposition simply and concisely so anyone that is not familiar with your product ecosystem can understand, you have a big problem in your hands.

This product seems like a simple scheduling tool, but your marketing jargon fluff piece says otherwise.

I would think that Quartz is more of a competitor than cron or Windows Scheduler to Prefect Scheduler.

How does Prefect Scheduler compare to something like Quartz?