I agree pretty strongly with the premise of this article. Unexpected or unnoticed bottlenecks and localized issues can lead to cascading failures.
A book I read more than a decade ago helped me learn about this stuff: Release It! by Michael T. Nygard. https://www.oreilly.com/library/view/release-it/978168050026... I am wondering if the information contained within is really dated now, but I am not really sure.
I believe that load-testing components in concert, identifying such spots, and then reproducing the problems in isolation and alleviating them may be a solution to preemptively find such issues. That does come with the cost of such work and assumes that one can use a workload that is representative of actual issue-causing workloads in production. I believe that this is a skill that people and teams can learn, but mistakes are easy to make. I also think one should be careful with using production data for privacy reasons and to avoid accidentally performing real-world actions with consequences during testing - another area of software development I feel is worth considering (conscious and well-designed separation of environments into development and testing and production segments, e.g.).
I also agree with the premise, and I also really enjoyed release it. IMO it holds up well!
I don't agree that everyone should spend time looking for potential problems and preemptively fixing them though. For sure some amount of this is worthwhile, but in my experience actual problems tend to trump potential future problems.
The challenge is that it can often seem like not a good use of time to fix the "small problem" that isn't really causing any harm right now (that's what makes it a small problem, really), it's just... messy. As OP says "we’ll likely spend some of our time fixing Small Problems that never would’ve amounted to anything." So it can be hard to justify spending this time.
But I find it helpful to use the analogy of: Keeping our workspace clean and tidy.
A messy disorganized workspace isn't on it's own necessarily considered a problem, it may not be visibly harming your output. But it can lead to problems. When you trip and fall on something you left behind. And it can make it harder to respond to problems when they happen, when you can't figure out what's going on because you can't find the tool you need quickly, or there's just so much in the way that you can't even see what's going on.
Similarly, fixing the thing in your infrastructure that's "not quite right" even though it's not yet causing a problem -- will make the big problems less likely. And make them easier to diagnose and solve when they do happen, because everything is just neat and tidy so there are fewer things to rule out as the problem, fewer things obscuring your view of the problem.
But you also don't want to spend all your time improving "not quite right" things that might never amount to anything, of course. One solution is just making a budget for it, we'll spend X amount of time a week/month "tidying up the environment", responding to small problems.
And this all is not, by the way, only relevant when you are "under load" or "at web scale." A tidy environment matters for any deployment.
3 comments
[ 182 ms ] story [ 813 ms ] threadA book I read more than a decade ago helped me learn about this stuff: Release It! by Michael T. Nygard. https://www.oreilly.com/library/view/release-it/978168050026... I am wondering if the information contained within is really dated now, but I am not really sure.
I believe that load-testing components in concert, identifying such spots, and then reproducing the problems in isolation and alleviating them may be a solution to preemptively find such issues. That does come with the cost of such work and assumes that one can use a workload that is representative of actual issue-causing workloads in production. I believe that this is a skill that people and teams can learn, but mistakes are easy to make. I also think one should be careful with using production data for privacy reasons and to avoid accidentally performing real-world actions with consequences during testing - another area of software development I feel is worth considering (conscious and well-designed separation of environments into development and testing and production segments, e.g.).
I don't agree that everyone should spend time looking for potential problems and preemptively fixing them though. For sure some amount of this is worthwhile, but in my experience actual problems tend to trump potential future problems.
But I find it helpful to use the analogy of: Keeping our workspace clean and tidy.
A messy disorganized workspace isn't on it's own necessarily considered a problem, it may not be visibly harming your output. But it can lead to problems. When you trip and fall on something you left behind. And it can make it harder to respond to problems when they happen, when you can't figure out what's going on because you can't find the tool you need quickly, or there's just so much in the way that you can't even see what's going on.
Similarly, fixing the thing in your infrastructure that's "not quite right" even though it's not yet causing a problem -- will make the big problems less likely. And make them easier to diagnose and solve when they do happen, because everything is just neat and tidy so there are fewer things to rule out as the problem, fewer things obscuring your view of the problem.
But you also don't want to spend all your time improving "not quite right" things that might never amount to anything, of course. One solution is just making a budget for it, we'll spend X amount of time a week/month "tidying up the environment", responding to small problems.
And this all is not, by the way, only relevant when you are "under load" or "at web scale." A tidy environment matters for any deployment.