HN Explorer

Ask HN: What do you use to manage chains of long-running jobs?

4 points by polm23 ↗ HN
For running chains of multi-hour jobs with dependencies I've always used Jenkins but always been unsatisfied with it. I've also heard of Luigi but haven't given it a shot yet. Is there anything else similar I should look at? I've had trouble finding a single name for programs like this to search with.

Luigi: https://github.com/spotify/luigi

Example jobs:

  - download a bunch of raw data from different sources
  - clean and transform raw data into a single format
  - push a lot of data into a DB
  - routine ML model updates
  - generate reports
These would have dependency relations, some would be short and some would be long, sometimes they fail (which may be important or not), etc...

5 comments

[ 2.3 ms ] story [ 20.8 ms ] thread
shoo
> For running chains of multi-hour jobs with dependencies I've always used Jenkins but always been unsatisfied with it

What in particular are you unsatisfied with?

What are you happy with that you wouldn't want to lose?

ditemis
The search term "batch processing" might give you some alternative solutions.

Jberet, a Java Batch API (JSR 352) implementation, is a batch processing framework and can be run in a standalone Java process or within Wildfy/Jboss application server. It provides a robust batch execution runtime, XML configuration for jobs, REST API for batch management, a lightweight web frontend and various data readers/writers. https://github.com/jberet/jsr352

eb0la
I use Talend Open Studio a lot for this kind of stuff.

It might not be the best tool for the ML part; but for moving data, and ssh'in automatically to servers and executing commands is really good.

I used it (a long time ago) to ssh to a set of Juniper routers, execute a command, parse the results (in XML), update some db tables, and execute another commands in other routers...

I really like the wait for file feature ( http://imgur.com/a/3kKLC )