Ask HN: Has anyone fully embraced an event-driven architecture?

For us, it's either a function that will retry the messages after some time, or manual intervention.

Our department recently said we need to move to even driven architecture for one of our processes that currently runs in a batch. They want us to load data into EMR from an S3 bucket populated by Kinesis. Their suggested implementation is simply to run the batch job more frequently instead of once a day... sorry guys, but that's not even driven...

I suggested maybe just setting a trigger on the S3 bucket and hook it up to Glue, since that would actually be event driven. They said 'no' because they don't want the load to EMR or Glue to run too frequently. I guess that makes sense (not that familiar with the ETL tech), but it sure doesn't make sense to call it event driven.

One of the benefits of this architecture for us is the ability to easily share information between services. We utilize SNS and SQS for a pub/sub architecture so if we need to expose more information we can just publish another type of message to the topic or if we need to consume some information then we can just listen to the relevant topic.

There are two big issues that I've run into while at this company. One is tracking down where events are coming from can be a big pain, especially as we are replacing services but keeping message formats the same. The other big issue is setting up lower environments (dev,qa,etc) can be difficult because you pretty much need the entire ecosystem in order for the environment to be usable, which requires buy-in from all teams in the organization

Events are a part of a greater whole. It's a tool that you can use to solve certain data flows, but not all data flows. When you start taking more liberty with the word "eventually," you are almost certainly in a realm where event-driven makes the most sense. CQRS is a pretty good example of using many architectures (including event-driven) under a single greater architectural umbrella, and the thought patterns it introduces you to are incredibly useful. But no architecture is gospel, not even close.

Any "pure" architecture is the tail wagging the dog. The problem comes first, the solution comes second, the architecture comes third.

It's great for processing data that goes beyond a single database call, data formatting and presenting something on a page.

If you're chaining multiple microservices together you've made a very sloppy/poor mans version of this. (People tend not to account for downed services, maintence updates, client durability, etc) When you bring in technologies like Kafka to orchestrate this, you'll end up with a more reliable system that you can fix if something goes wrong. This more changes the way you think about data and how you present it. Also, it'll increase your uptime because your service's SLA is isolated from what you're processing. (Your service and the persistent storage that is storing state is what people see.. the data being out of date is something you should account for)

Schema changes: Generally you don't have that big of a deal because multiple applications get started at once. You should have system level tests to catch that before you go out. Also, application smoke tests help as well. As long as you picked a durable message queue with a framework that'll crash on error, you can fix that, bring up the fix and continue processing through.

Dead letter queues: It's more about how you architect more than anything. This is something you should plan for.

"fully embraces" / commits - not a very wise thing to do. Event driven arch is one of many tools at your disposal. It is awesome for some things and not so much for others. You can't use single tool / approach for everything and expect best results.

The architecture made sense since events (new trades or quotes) dictate a host of downstream activites, which often need to be near-real-time reduce divergence.

We used Kafka for event-driven micro services quite a bit at Uber. I lead the team that owned schemas on Kafka there for a while. We just did not accept breaking schema changes within the topic. Same as you would expect from any other public-facing API. We also didnt allow multiplexing the topic with multiple schemas. This wasn’t just because it made my life easier. A large portion of the topics we had went on to become analytical tables in Hive. Breaking changes would break those tables. If you absolutely have to break the schema, make a new topic with a new consumer and phase out the old. This puts a lot of onus on the producer, so we tried to make tools to help. We had a central schema registry with the topics the schemas paired to that showed producers who their consumers were, so if breaking changes absolutely had to happen, they knew who to talk to. In practice though, we never got much pushback on the no-breaking changes rule.

DLQ practices were decided by teams based on need, too many things there to consider to make blanket rules. When in your code did it fail? Is this consumer idempotent? Have we consumed a more recent event that would have over-written this event? Are you paying for some API that your auto-retry churning away in your DLQ is going to cost you a ton of money? Sometimes you may not even want a DLQ, you want a poison pill. That lets you assess what is happening immediately and not have to worry about replays at all.

I hope one of the books you are talking about is Designing Data Intensive Applications, because it is really fantastic. I joke that it is frustrating that so much of what I learned over years on the data team could be written so succinctly in a book.

It works alright (like you I hated it all before, coming from more amateurish implementations). It's slow to change (adding a new event type can take years before it works everywhere), failures can happen - for instance if the message parsing library has a crushing bug and an extendable attribute zone has a poison pill that never appeared before - well nothing you can do but manually editing the event source.

What it brings us, I suppose, is that every micro service is single threaded, all events are well-recorded , we use multicast to transfer them from one sequencer to all consumers so we just need good routers and TCP-level message building - it's very barebone to keep it fast, extensibility for us is mainly on adding more services around a core stream that we don't really need to change all that much (we do a lot of regulatory validations, data analysis, the odd scale out for a round-robin compatible process - not all of them are, some need to see all events, for instance for cumulative exposure calculation - client stock exposure on several markets for risk-based decisions).

It also avoid latencies like in state-request based systems, since each service will build its own state machine. We make a lot of money on this system, so we can hire hundreds of people around the world to maintain it.

At this point I dont see how to do it better (5ms round trip to the client if low validation, 100ms if crossing seawater to an exchange with short selling validations) without events, but I know very well that doing it for simpler flows that are not latency-constrained will probably result in heavy cost and low gain. I would never recommend it for people who aren't already struggling with a fully-fledged business implementation that makes money they want to accelerate.

The problems we face are:

- it's slow to evolve the very core of the system

- we need perfect ordering, there will always be time wasted at the sequencer to transform unordered "commands" for the various services into perfectly ordered events

- testing and debugging is an art that takes time to acquire: I can now, but the task seemed daunting when I started - how to spin up the minimal surface of services to make a valuable replay, how to make static configuration reproduce production's behavior exactly so that all services behave the way you want to reproduce if you're investigating a sequence-based issue (rather than a function-based issue)

- it takes up to a year for a new Java dev to get productive on such an exotic mindset, but it's also because we do no intraday malloc since we cannot afford a GC in the middle of a client order

- management cannot understand why they can't cut cost using the cloud, virtual machines, vendor databases etc. Even in a company that makes billions over 20 years on this system, we still can't explain it in a way that sounds valuable vs its cost. Because its cost probably is extremely high, and can't be outsourced by hit and run consulting managers before being brought in-house again. So we're not like the most popular dev, we're the slow and expensive ones :(

Basically everything that makes fully event driven architectures difficult is ameliroated because the only consumers are myself and my wife, and we literally built up the whole system. Something appears to be locked up? There's a system of watchdogs to kill stuff, all hardware has been designed to fail off into manual control, and we can pick the pieces up at our leisure like when anything else in the house breaks. The last will and testament messages in MQTT are really nice for at least reporting hard failure conditions.

I'll be the first to admit that I would not look forward to productizing it and supporting someone else's house (to the point that I'll probably never do that). It's so easy for messages to make their way into the bit bucket when setting up a new subsystem, and everything is so loosely coupled because of the event system it's almost like it's all "stringly typed". And being both software engineers, we sort of relish in how awful the UI is, even using 98.css.

Unless the scale is an issue, nowadays I always prefer a more state-full approach when possible.