From my experience, debugging monoliths is much easier. Microservices a bit hard (mostly networking). Event Driven Architecture with microservices is even harder. My company's brief stint into building one out turned into a disaster of trying to isolate wtf was going on when there was some issue.
Now that could have been the engineers/architects fault, but unless my team grows, I would hate having to deal with it again.
I think it depends on the isolation of each part. If the incoming data is fine, then you have the problem narrowed down already. If it isn't, then you have to figure out why it was sent in the first place, probably by detecting something wrong in the sender at run time so you can see what led to the bad message.
I don't hate debugging in evented/microservice codebases, but it is more complicated to get a good debugging setup in place w/ distributed tracing. But after you do that you're often in a better place with logging, so, it's tradeoffs. It's all tradeoffs in fact, I'll take a monolith where it's appropriate for the space and I'll take evented/microservice architectures when the monolith pattern is no longer a good investment for the problem space. All these things exist for good reasons, don't be an emotional investor [of your own time/opinions] ;-).
Not really. It's sometimes it's easier than debugging monoliths - just log the incoming message, log outgoing message. And you can narrow down the problem just as quickly as in a monolith.
Sometimes when you inspect logs from a single component it feels like trying to solve a problem with the forest by inspecting all the ants in it. Lots of mostly irrelevant small detail without any way to place it in the big picture.
which is why you want more tracing and less logging.
log messages are always an art and I myself am a poor artist when it come to predicting what is both necessary and useful enough to log without overall increasing the log noise.
Tracing will show you the distributed behavior, but the best log, at most, can only show you the current state of information on a single service on a single node even when you aggregate. Tracing also works hand in hand because the correlation id should associate with the relevant log message when you do decide to drill down.
> Probably every technical decision maker pushing for microservices knows the perils of distributed debugging
That doesn't line up with my experience. I think humans have a pretty strong tendency to take for granted the advantages of a current situation when looking at something that solves a very specific new problem. "The grass is always greener"
"Hating" something of course doesn't matter, what does matter is response time in being able to debug production systems, and developer efficiency, and satisfaction / employee retention. So it's not orthogonal, imo, as it's very likely correlated with an efficiency loss.
Right on. It takes quite a bit of experience to get to the point where you can sort of look into the future to think about what living with the new system will be like, once its warts and shortcomings are known. The time-sucking problems usually won't show up during a PoC where everything looks beautiful and easy.
My experience with debugging with RabbitMQ was not bad, but we had to do a bit of work to ease the process. For example, we configured error queues where processors publish error log and input message information, which is enough to reproduce the bug and understand what happened.
This isn't a sign that one architecture is better than the other. It only indicates that the tools are more mature for debugging monoliths than distributed architectures (including event-driven) for tracing causality and data flow through the whole system.
The fact that there's great advantages to distributed architectures, yet they're harder to work with, is a signal that there's a market opportunity for better tooling around working with them.
I think it comes down to taking debugging, monitoring, and alerting seriously. I doubt any team would be better off with a distributed system if they don't spend a serious amount of time setting up logging, monitoring, tracing, making it easy to debug, having automated alerts, etc. Its what separates a great coder from a good engineer and if you don't have the latter on hand, you should steer far away from building systems like this.
We've got a couple of these kinds of applications where I work. All built on RabbitMq / MassTransit. Without smart logging it would be real frustrating to track down issues. You could put breakpoints everywhere locally, but that's a pain.
Tools like Seq are really helpful during development. ELK stack / splunk / log aggregator X are almost required on the non-local environments.
Splunk aggregation is a life saver. If you assign a unique id to each request, you can trace its path thru it's entire lifecycle, including asynchronous post-tasks. This can also help see average delay latencies between a sync task and its associated asynchronous post task/s.
Every time I work somewhere I have to play shepherd and ask the very basics:
- Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments
- Have you thought about all the new problems that might happen: queue sending to dead endpoints, circular queue problem, queue being restarted somehow (e.g. deploys) and losing messages?
- If the app fails post-queue, not surfacing the message to the user, do you have a plan to ensure somebody in engineering sees and fixes that error? And then goes back and remediates the broken request(s)?
- Have you prepared code/logs to do distributed tracing?
- If there's a dispute a week from now whether Joe didn't get an email because of a problem BEFORE or AFTER the queue, will you be able to tell from the logs?
Many powerful engineering abstractions (threads, async, services) require one notch higher of engineering talent and allows for all sorts of new failure paths. The tradeoff must be taken very seriously. Most places I have worked at adopted complexity too soon.
Fully agree here. Catastrophic failures can slip in silently on both ends of the curve -- if Joe doesn't get an email OR Joe gets too many emails, it's still bad. In my experience, beefing up logging to be able to do distributed tracing is an absolute must to prevent debugging pain.
- Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments
Azure and AWS and every other service that provides a Service Bus service guarantees up-time. Their SLA should be your customers SLA.
This is a subcategory of "app fails post-queue." Suppose you build a happy working queue, and someday somebody releases a new version of the queue consumer that isn't backward compatible and therefore is broken. How many messages will be lost on production before this caught? Do you have a way to recover those messages?
> circular queue problem
This is the queue version of an infinite loop. Suppose you have an infinite loop in your code, you'll crash the app but catch it very quickly.
Suppose queue message X calls a function which generates ANOTHER queue message X (infinite loop). This will be VERY HARD to catch and slow down the queue system progressively until its overwhelmed (likely only caught on prod).
Logic spread out as different code parts that consumes events and then creates new events can be very hard to follow. A simple loop might be obvious but when part A creates event B and part C reads event B and then creats a new event that part A will read it is harder to find. It may be many steps and also some logic that makes it only happen for specific payloads.
Great explanations by the way but I try to avoid having too much logic spread out over different events.
It can be very hard to know the state of a piece of information within your system unless you fanatically log its state at all times. And even then, you might have to look through lots of distributed logs to find that. Of course, logging the state of the world at every step also creates tons of noise.
At a previous gig, we had a processing workflow where an entity might move through any of 20 processes tied together through queues. It was very, very difficult to track down problems with things dropping out, things getting stuck, etc.
Imagine if you flipped on debug logging and suddenly your queueing system gets DDoS by log messages, causing more errors and cascading effects, taking down your product. It’s always a good idea to separate the two.
> where an entity might move through any of 20 processes
To me that's a design smell. The processes should be making requests of the entity via messaging. Each process should result in the entity emitting a change of state. That change of state should be the trigger for subsequent processes.
The timing of this article couldn't have been more painful.
I'm currently working on solving a problem with an insanely over complicated setup (for the task at hand) that is build by another engineer who has since left the company.
It's a cluster of 3 virtualised machines running docker swarm where a RabbitMQ instance ties 40+ worker pods together. Once every 5 days or so the connection between RabbitMQ and (some of) the worker container stops working, causing the worker to crash and the queue message is lost.
We are talking like 5 layers of virtualization and/or abstraction. It's impossible to debug. I honestly don't know how to explain this to my customer.
Rabbit supports the choice of durable exchanges, queues, and messages. There are (tunable, negotiable) performance penalties in terms of latency and throughput.
We are partly in agreement regarding f/t. Logging comes into play when digging into recurring failure (bugs). Fault tolerance will not save you from bugs and bug hunting.
[p.s. to add, performance tuning/trouble-shooting also require instrumentation. Instrumentation would appear to be a fundamental (and useful) capability.]
it sounds like this setup has proven very successful in helping your predecessor get hired (away from the company), I wonder if that was an implicit or an explicit requirement when they were designing the system
I've found it very helpful in distributed environments to ensure that you are giving the incoming request an extra "Request ID"-style header. Make sure this header is propagated everywhere, and logged everywhere. Makes debugging much, much easier. Still gets hard at scale, which is why tools like New Relic have various "distributed tracing" features now.
I forget the exact quote, but it goes something like "the best distributed system is one you didn't build". Obviously there are reasons to build them that become inevitable once a system needs to grow beyond a certain size, but if you can put that off for a while, it's a win.
To your point...even if you do want eventing, you can do it inside of a monolith and not involve the network at all, which will save a good amount of pain.
Not really, it's tougher, but if you're bringing queues into your workflow then it should be because you need it. You are having to deal with scaling, resiliency, parallelism, faster response time, etc. The benefit then out weighs the debugging overhead. But at that point you have a distributed system. If you have a Distributed system, you should have centralized logging and distributed tracing too so you can debug easier. Without those, you are going to go through the pain.
Of course but you're often trading debug-ability over scalability whenever you're picking evented architecture. I do think tools can help here but it's just the nature of this architecture. You're reacting to 'events' instead of following procedural orders.
we just finished implementation of an event sourced/CQRS solution using kafka.
Yes, you need to have monitoring etc, but the testing and debugging were substantially easier because of how we broke down the "services". For each major entity (or aggregate) there was a service that subscribed to a number of command and event topics, it produced output to an event topic for the aggregate.
We had FSMs for each of the aggregates, documenting the effect of each potential command or external event and the change of state and the resulting state changes (events) and/or commands.
The architectural constraint meant that the infrastructure was the same for each aggregate, the testing of each was independent and could be mocked easily using topic producers.
So as opposed to the "Big Ball Of Mud" we have a monitorable infrastructure (kafka + alerts/stats sent to a statsd integrated with AWS Cloudwatch), we have individual aggregate processing that only respond to incoming commands or events and have defined outputs for each potential incoming command/event.
Much much easier to design, develop and debug.
But the trick is at the start (like anything else). Analyzing the domain to determine the entities/aggregates, modelling the externally generated commands, modelling the FSMs for each aggregate etc.
Queue driven systems really fascinate me, coming from a chemical engineering background I can't help but to see parallels to fluid dynamics and all that difficult math that comes from their analysis.
I've always wanted to create some type of monitoring system that displays the entire system in that vein and then model or using control theory.
55 comments
[ 3.1 ms ] story [ 98.6 ms ] threadNow that could have been the engineers/architects fault, but unless my team grows, I would hate having to deal with it again.
I feel like the more shortsighted/incentivized by sheer work volume a person is, the more they're into monorepos...
Can you debug a forest by inspecting a single tree?
Tracing will show you the distributed behavior, but the best log, at most, can only show you the current state of information on a single service on a single node even when you aggregate. Tracing also works hand in hand because the correlation id should associate with the relevant log message when you do decide to drill down.
Probably every technical decision maker pushing for microservices knows the perils of distributed debugging. They have some weight - just some.
That doesn't line up with my experience. I think humans have a pretty strong tendency to take for granted the advantages of a current situation when looking at something that solves a very specific new problem. "The grass is always greener"
"Hating" something of course doesn't matter, what does matter is response time in being able to debug production systems, and developer efficiency, and satisfaction / employee retention. So it's not orthogonal, imo, as it's very likely correlated with an efficiency loss.
The fact that there's great advantages to distributed architectures, yet they're harder to work with, is a signal that there's a market opportunity for better tooling around working with them.
We've got a couple of these kinds of applications where I work. All built on RabbitMq / MassTransit. Without smart logging it would be real frustrating to track down issues. You could put breakpoints everywhere locally, but that's a pain.
Tools like Seq are really helpful during development. ELK stack / splunk / log aggregator X are almost required on the non-local environments.
Every time I work somewhere I have to play shepherd and ask the very basics:
- Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments
- Have you thought about all the new problems that might happen: queue sending to dead endpoints, circular queue problem, queue being restarted somehow (e.g. deploys) and losing messages?
- If the app fails post-queue, not surfacing the message to the user, do you have a plan to ensure somebody in engineering sees and fixes that error? And then goes back and remediates the broken request(s)?
- Have you prepared code/logs to do distributed tracing?
- If there's a dispute a week from now whether Joe didn't get an email because of a problem BEFORE or AFTER the queue, will you be able to tell from the logs?
Many powerful engineering abstractions (threads, async, services) require one notch higher of engineering talent and allows for all sorts of new failure paths. The tradeoff must be taken very seriously. Most places I have worked at adopted complexity too soon.
Azure and AWS and every other service that provides a Service Bus service guarantees up-time. Their SLA should be your customers SLA.
> queue sending to dead endpoints
> circular queue problem
This is a subcategory of "app fails post-queue." Suppose you build a happy working queue, and someday somebody releases a new version of the queue consumer that isn't backward compatible and therefore is broken. How many messages will be lost on production before this caught? Do you have a way to recover those messages?
> circular queue problem
This is the queue version of an infinite loop. Suppose you have an infinite loop in your code, you'll crash the app but catch it very quickly.
Suppose queue message X calls a function which generates ANOTHER queue message X (infinite loop). This will be VERY HARD to catch and slow down the queue system progressively until its overwhelmed (likely only caught on prod).
Interesting point about things getting caught on prod, no amount of stress testing can sometimes reproduce these finicky bugs :)
Great explanations by the way but I try to avoid having too much logic spread out over different events.
At a previous gig, we had a processing workflow where an entity might move through any of 20 processes tied together through queues. It was very, very difficult to track down problems with things dropping out, things getting stuck, etc.
To me that's a design smell. The processes should be making requests of the entity via messaging. Each process should result in the entity emitting a change of state. That change of state should be the trigger for subsequent processes.
I'm currently working on solving a problem with an insanely over complicated setup (for the task at hand) that is build by another engineer who has since left the company.
It's a cluster of 3 virtualised machines running docker swarm where a RabbitMQ instance ties 40+ worker pods together. Once every 5 days or so the connection between RabbitMQ and (some of) the worker container stops working, causing the worker to crash and the queue message is lost.
We are talking like 5 layers of virtualization and/or abstraction. It's impossible to debug. I honestly don't know how to explain this to my customer.
We are partly in agreement regarding f/t. Logging comes into play when digging into recurring failure (bugs). Fault tolerance will not save you from bugs and bug hunting.
[p.s. to add, performance tuning/trouble-shooting also require instrumentation. Instrumentation would appear to be a fundamental (and useful) capability.]
Yes, you need to have monitoring etc, but the testing and debugging were substantially easier because of how we broke down the "services". For each major entity (or aggregate) there was a service that subscribed to a number of command and event topics, it produced output to an event topic for the aggregate.
We had FSMs for each of the aggregates, documenting the effect of each potential command or external event and the change of state and the resulting state changes (events) and/or commands.
The architectural constraint meant that the infrastructure was the same for each aggregate, the testing of each was independent and could be mocked easily using topic producers.
So as opposed to the "Big Ball Of Mud" we have a monitorable infrastructure (kafka + alerts/stats sent to a statsd integrated with AWS Cloudwatch), we have individual aggregate processing that only respond to incoming commands or events and have defined outputs for each potential incoming command/event.
Much much easier to design, develop and debug.
But the trick is at the start (like anything else). Analyzing the domain to determine the entities/aggregates, modelling the externally generated commands, modelling the FSMs for each aggregate etc.
I've always wanted to create some type of monitoring system that displays the entire system in that vein and then model or using control theory.
Has anybody seen a project that does this?