I've shortened the title just a bit due to the character limit.
72 hours of production downtime is /insane/ to think about and with such a brass attitude coming from the team as well.... I'd be inclined to go to the board and ask for additional funding to start a new team/begin knowledge transfer ASAP and start winding down the old team. I'd be curious to know why the CTO's isn't being grilled about the downtime and if they even have a CAPA process to prevent this from happening again?
Additionally, one of the commenters in the thread also points out just how important devops as a /culture/ is compared to just treating it as something to be bolted on at a later point in the organization life. I've worked with too many devs that seem to think their involvement ends when they throw the code over the proverbial "devops fence".
I'm curious how the compensation is structured. For someone with equity in the business a 72 hours downtime is horrible.
But if the devs are making below market rate, and their compensation is unrelated to the business' performance, it's a little unrealistic to expect them to be going above and beyond for on-call duties.
Start by firing the guy who pushed the code on Friday Night. Then email the rest of the team and say anyone who wants a job to just show up 9AM Monday.
And then tell them all pushes to prod go through you for a time.
I have had to do this in a few roles.
I dont understand the “break fast and fix things” approach for a lot of companies.
For me uptime and security are king. Id rather maintain a reliable cadence over just invoking chaos. Especially when everyone is siloed. It becomes managements role to ensure rollouts are planned so outages can quickly be troubleshot or rolled back.
Now i dont work for a company like netflix. But discipline can scale in my opinion. There is a fine line between paralysis by analysis and break fast and fix.
5 comments
[ 2.9 ms ] story [ 20.5 ms ] thread72 hours of production downtime is /insane/ to think about and with such a brass attitude coming from the team as well.... I'd be inclined to go to the board and ask for additional funding to start a new team/begin knowledge transfer ASAP and start winding down the old team. I'd be curious to know why the CTO's isn't being grilled about the downtime and if they even have a CAPA process to prevent this from happening again?
Additionally, one of the commenters in the thread also points out just how important devops as a /culture/ is compared to just treating it as something to be bolted on at a later point in the organization life. I've worked with too many devs that seem to think their involvement ends when they throw the code over the proverbial "devops fence".
But if the devs are making below market rate, and their compensation is unrelated to the business' performance, it's a little unrealistic to expect them to be going above and beyond for on-call duties.
Yikes.
I have had to do this in a few roles.
I dont understand the “break fast and fix things” approach for a lot of companies.
For me uptime and security are king. Id rather maintain a reliable cadence over just invoking chaos. Especially when everyone is siloed. It becomes managements role to ensure rollouts are planned so outages can quickly be troubleshot or rolled back.
Now i dont work for a company like netflix. But discipline can scale in my opinion. There is a fine line between paralysis by analysis and break fast and fix.