Ask HN: How do you handle long-running workflows at your company?

In publishing/media:

Some workflows are shorter than others, but in the journalism side the workflows tend to bottom out at a day and max out at a few months (for the workflow, but is ultimately dependent on the weight of the story)...

Most of that is handled above the technology, mind.

The exploration for the right tool(s) is ongoing. I've been leveraged to build one but the status of that clandestine project is in flux to put it lightly. Not sure if I can elaborate on that right now.

Currently, the needs and preferences vary so much that there are many different services used, but the company is seeking to centralize some efforts (like content generation and management) and externalize others (like distribution).

We basically email stuff around and then when it gets stuck somewhere follow up with another email / conference call to move it forward again. If it keeps getting stuck or doesn't move it's obviously not an important process so it falls out of the loop.

kafka streams and spark jobs.

Our system is based on the Camunda process engine (in a Java EE environment). There's a central process server (or cluster) running the process engine, with events to start process instances.

Workflows are defined using bpmn and then executed by the engine. Errors are reported to the process engine as "Incident", which then show up in the management ui/apis. These can be retried any number of times.

We also have an older system based on Carnot/Stardust/IPP. This one used JMS messages everywhere.

"State machine" was the easiest for simpler stuff. I put it in quotes because it feels like one, but probably isn't.

Map out each state of your workflow, and having errors give the option to fix immediately, try again, or revert to a previously known-good state. You likely want to start with a 10,000ft view of the workflow and then work on each of those steps as an independent unit and add all of their intermediate steps (on and on until you reach the bottom of the recursion).

This gives you a good opportunity to break things up into microservices that completely handle individual steps if they are big and detailed enough.

P2P is hardest because you will likely need to code something to determine who should decide to move things to the next state (simple majority? one person elected?) and keep track of consensus between all parties.

Orchestration is easier because there's usually one person, one role, or one security claim in control at a particular step and changing who can advance the state at each step is pretty easy as well.

All of this was mostly for the goal of really easy unit testing.

But note that whatever backing data store you use can be changed by any developer unless you code all of the business rules there, too. Many people do not like doing this though because it's not as easy as all of the unit testing frameworks, debuggers, and IDEs we have for code.

The challenge is that you need to know the workflow completely and that will very likely involve talking to a lot of people and the chances you will miss one or two edge cases is high. The counter to that challenge is that as developers building a product that saves time/money, you can bend the workflow to make it easier to code and sometimes eliminate those extra steps (literally, we had someone copying and pasting stuff to 'make it work', so of course we can automate that).

Saving known-good states can also be challenging depending on what you're doing, but if you need change history or diff'ing in a user-consumable form, you'll have to do that anyway. If you get this right, it can save your users a lot of potentially lost work and headache if a bug gets past unit testing.

Once everything is modular, logging isn't too difficult either.

I've used Redmine with the Checklists plugin for this. Each thing that needs to be done is a redmine issue, and each issue can have a checklist. As team members check off items on the list, the issue logs who/what/when and then the user can assign the next person in the chain to the issue. At the time the checklists plugin didn't include templating functionality (not sure if it does now), so I rolled my own using the Redmine REST api and some PHP.

Hardest part was getting managerial support; they really liked paper.

We have developed a custom workflow system in PHP for our company (academic text editing and related work). Back then (I was not directly involved from the start) none of the out of the box solutions fit our criteria, and it made more sense to just build a bespoke, custom fitted system. Workflows range from a few days to a month+, with no technical upper limit enforced, as far as I know.

We also don't need a huge throughput, so having something super-optimized was not a large concern.

I had been looking at using BPMN and an implementation of Camunda as a reasonable goal, but I never found an implementation of running a BPMN service that I liked in the time I had allotted. In the workflow each item is essentially a ticket so you end up with concurrent tickets in the state machine. It also has timers to generate events so you can have that monthly event start and trigger some other actions, and it also includes failure paths.

The only tool I have found that checks all those boxes (and then some) is Airflow. I liked it so much that I went to work for Astronomer.io, which is building a managed and on-prem solution to Airflow.

It's not the perfect tool, but we are striving to make it better.

IIS, AppFabric, Windows Workflow Foundation services (WF). Leverage there for orchestration, persistence, error handling, etc. Considering the demise of AppFabric, do you mind if I Ask HN: How would you handle these long-running workflows in the long-run (and keep IIS and WF)?

A custom layer built on top of Celery that allows for better monitoring and dependency management, amongst other things. Monitoring, particularly of failure is pretty ok in Celery anyway.

The whole thing can generate its own graph by inspecting dependencies, and we use dagre to draw pretty process workflows with status, interactions and monitoring.

On some of our Ruby workflows We use Sidekiq Pro which has scheduled and batched jobs. The batched jobs is neat because it has a callback feature that you can use for starting additional steps / workflows. We monitor/alert on progress with statsd, datadog, and the sidekiq ui.

Our main bus for microservices is a RabbitMQ cluster. Most services have their own isolated write store and read store (which might be a true read store, or just a db replica).

Long running jobs are a rarity, so we usually spin up a new RabbitMQ cluster and services, but tie those services back to the main write/read stores. This allows regular operations to still occur, but we can monitor the bulk process and commit resources to it in a more isolated fashion.

Errors end up in error queues in Rabbit, and can be dumped back in to be reprocessed if appropriate (or just ignored if it's a side effect we don't care about).

Once it's setup and running, it works well enough. Spinning up a new rabbit cluster and service instances is currently manual, but since we've moved to Kubernetes I'm hoping this can be automated almost entirely.

We are currently using Activiti (a fork of jBPM) for a client project. Tooling is pretty shoddy for changing the workflow, but it works.

pipefy is kinda cool. we don't use it but i've been looking at implementing some of our processes in it.

There is no obvious solution right now. That's why we are building Zenaton (I'm cofounder). It's in closed beta by now, but you can have a look at the documentation (https://zenaton.com/documentation) and also read some use cases (https://medium.com/zenaton). Zenaton provides a very simple way (in your own programming language) to orchestrate background jobs

Check out Luigi (Python -- https://github.com/spotify/luigi).

I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.

Airflow jobs in the backend.

Nomad just does it for you(mostly): https://www.nomadproject.io/docs/job-specification/parameter...

Uber wrote its own framework: https://github.com/uber/cadence

We use (as well as sell/implement/customize) an Enterprise Content Management system that has very robust business automation capabilities.

https://www.laserfiche.com

Two options we use, database used as a queue for granular out of process work. If something errors, we'll get a notification for that one record, but the rest will keep processing.

For stuff that we don't need such granularity/replay, we use Amazon's SNS event framework to trigger different APIs.

Sometimes we do a combination of those, an SNS event triggers a lambda that puts a record in the database queue, which gets picked up by a job engine and raises an SNS event that hits an API that sets a record to available.

We let the web application do it and pray the web server does't croak mid job which it usually does.

I'm the CTO at ProcessMaker and so I might be a little biased. Our customers use our ProcessMaker BPM product if the workflows require human intervention through forms/email other interactions. The reporting tools assist in monitoring and dealing with circular chains.

If you are a developer and want to develop your own system around a workflow engine, we also have www.processmaker.io which is a workflow engine in the cloud. So all the infrastructure hassle is taken care of for you and you communicate via an api to build out your workflows and execute them. I feel like that's better described as an orchestration engine however it supports task assignment to people. An approach like this works well with microservices since it can act as a microservice orchestration engine with a more human workflow approach.

Both of these approaches can be long running (some customers have year long processes running).

Let me know if you want to know more details, happy to share.