HN Explorer

Ask HN: Scheduled events in an event sourced system

3 points by crowdswisdom ↗ HN
I've been learning a bit about Domain Driven Design, CQRS, and event sourcing. The domain I'm working with (at least parts of it) seems like an ideal use case of event sourcing.

But there's a part of it I haven't been able to wrap my head around: How to set up automated/scheduled tasks.

I'm most familiar with Python and Node/JavaScript, and that's what I'm planning to use as a starting point for the back end of my system.

In the past I've set scripts running on a crontab, and scheduled tasks leveraging Celery, RabbitMQ, and Redis.. but I've never built something with scheduled data fetching that fed that data into an event sourced system. Using the tools I know don't feel quite right.. like they're the nail to my hammer and I'm missing something.

To simplify: Basically I have one service that'll rely on an external data source (the data I need is available over a RESTful API), and another service that uses the resulting state of that data to create events in its own context.

Does anyone have experience with this sort of architecture? Could you offer any advice or point me at some resources?

Thank you in advance!

5 comments

[ 0.25 ms ] story [ 17.2 ms ] thread
giantg2
We're trying to switch to an "event driven" design. I think it's basically in name only. We will still be running several batches as part of the overall process (cron, SQS, etc). So I can't help you, and apparently nobody at my company can either.

There could be some other options. It depends on what the task needs to do and where it is in the process. There are triggers one could use, push style notifications from a prior step, etc. It could even make sense to keep a certain task as batch depending on how it fits in with the other processes.

crowdswisdom
Thanks for the reply and the thoughts. Interesting to hear that you're having a similarly hard time finding answers.

I've come across these concepts of "sagas" and "process managers" that sound maybe like what I'm trying to do. But it seems to me like all it might need to be is a background process that issues commands.

giantg2
There are tools for process maps that can execute code at specific steps based on criteria. I spent several years working with IBM Filenet process maps. There are other, similar tools. You can also build a rules engine in a DB with a either a background process checking those conditions or potentially triggers tied to the data (depending on the DB).

There certainly are answers to be had. I just believe that in this corporate world, people understand that it's best for their career to call what they work on by some fancy new name so they can look good and make more money.

ttymck
Maybe I'm not the right person to answer this, but I think your question may be too vague.

I am failing to see what the friction is between scheduled tasks and an event sourced system. If a customer were logging in every day at 10am (on a schedule) and issued their own events based on some external state, your system would be none the wiser to the "scheduled" nature of their behavior.

yuppie_scum
Kubernetes Jobs are stable as of 1.21 or so