HN Explorer

12 comments

[ 3.7 ms ] story [ 37.6 ms ] thread
rowan_mcd
Very familiar with this problem. This process has happened at my last job and is just starting at my current job. Distributing the issues by team makes a ton of sense, but in my experience, it only gets so far. It’s always been unclear how to handle, and scale, the remaining 10% of on call burden.
mijoharas
Is there a definition for the "NOC" acronym that the author uses throughout? I can't see one, and it's not immediately clear from context to me.
jandrewrogers
From context, I think it might be Network Operations Center, which was a common acronym at one time.
x0x0
network operations center
[deleted]
(comment deleted)
afandian
I do wish posts like this had absolute numbers (or did I miss some?). Is this a monolith served by a team of 5, 15 or 50?
abdouls
I agree with the ownership part. As part of my team (devops), we try our best to help with alerts and incidents, however, without ownership we are limited.

We do not know a product or a component better than those who own it or code it.

mbrodersen
I am so looking forward to all the micro-services disaster stories guaranteed to appear in the future. A friend of mine is already deep in a 20 year old micro-services mud pile. If you think a spaghetti monolith is bad, just wait until you encounter a 100+ micro-services spaghetti monster. My WTF record has reach dizzying now heights just listening to my friend talking about a few of the problems he encountered working on it.
abraxas
I left my last job as a senior architect mostly because of microservices ran amok. I decided that there was little I could do to move the needle and since the microsevice mania drove them also to having close to a hundred of those things there wasn't a way to make a meaningful contribution other than tell them to throw it all in the garbage and start over with a vastly simplified architecture.
mbrodersen
Yep not surprised. The reality is that most micro-services only work OK today because they are basically a complete rewrite of the previous system or a brand new system. ANY brand new system (including monoliths) works perfectly fine because it is all new and everything is in the developers heads. Just wait a few years and the system will turn into not only spaghetti but distributed spaghetti as new developers join the team and the original writers of the system leave.
abraxas
It may be fine to route the alert to the code owner but yet another gotcha of the microsevice architecture is that a localized alert will often issue a cascade of failures and the component that alerts may or may not be the one that needs a fix nor is it necessarily the one where the problem can be mitigated.