Ask HN: How do you handle duplicate side effects when jobs, workflows retry?

You proxy those api calls yourself and have idempotency to cover you for those APIs that don’t have it. If you architect it right you won’t have more than a ms latency addition. You can avoid the race condition issues by using atomic records so if something else tries they’d see it’s in progress and exit.

Use something like Temporal

I think the answer is probably like most things: it depends.

- If the external service supports idempotent operations, use that option.

- If the external service doesn't, but has a "retrieval" feature (i.e. lookup if the thing already exists, e.g fetch refunds on a given payment), use that first.

- If the system has neither, assess how critical it is to avoid duplicates.

Idempotency is the only sustainable answer here. Whether it's at the database level using unique constraints or implementing idempotency keys in your API headers, you have to design for the 'at-least-once' delivery reality. I usually implement a 'processed_requests' table that stores the unique ID of the job. Before the worker executes any side effect (like a payment or email), it checks if that ID exists. If it does, it skips the execution and returns the previous result. It adds a bit of latency, but it's much cheaper than dealing with double-billing or corrupted data

This is one of those problems that gets significantly harder when your system spans multiple runtimes or platforms.

A few patterns that have worked well in practice:

1. Idempotency keys at the API boundary — every side-effecting call gets a client-generated UUID, and the receiver deduplicates. Simple, but think carefully about the TTL of your dedup window.

2. Outbox pattern — instead of directly calling the external service, write the intent to a local "outbox" table in the same transaction as your state change. A separate process polls the outbox and delivers. Debezium + CDC makes this quite clean.

3. For cross-system workflows: treat the saga orchestrator as the single source of truth for step completion. Each step checks its completion status before executing, so steps must be idempotent OR the orchestrator tracks state.

In practice, designing for at-least-once delivery + idempotent receivers is more reliable than trying to achieve exactly-once through distributed coordination. Exactly-once across system boundaries is effectively a myth outside of systems that support two-phase commit (and even then it's fragile).

Retries become dangerous when workflows aren't designed to be idempotent from the beginning.

What helped us was treating every job execution as replayable and attaching a unique operation key instead of relying on execution state alone.

Otherwise retries silently create data inconsistencies that only appear much later.