It’s fascinating how often major incidents like this go:
change in production → unforeseen consequence → alerts go off → issue is found → fix is applied → things seem to stabilize (this all takes 15 minutes) → different issues pop up, caused by the quick fix → prolonged work needed to definitively bring stability (this usually takes anywhere from an hour to much longer)
Perhaps the lesson learned is: unless you’re 100% sure, maybe consciously consider your “quick fix” for 10 minutes to really think through its consequences.
1 comment
[ 4.1 ms ] story [ 12.2 ms ] threadchange in production → unforeseen consequence → alerts go off → issue is found → fix is applied → things seem to stabilize (this all takes 15 minutes) → different issues pop up, caused by the quick fix → prolonged work needed to definitively bring stability (this usually takes anywhere from an hour to much longer)
Perhaps the lesson learned is: unless you’re 100% sure, maybe consciously consider your “quick fix” for 10 minutes to really think through its consequences.