How to manage oncall as an engineering manager?

The best way to manage on-call is to not have on-call. On-call means the organization is understaffed. Hiring new positions to handle off hours, will solve the problem. Good luck.

Without knowing the scale of company you're at it's hard to give advice

At Microsoft I headed Incident Count Reduction on my team where opex could be top priority & rotating on call would have a common thread between shifts through me (ie, I would know which issues were related or not, what fixes were in the pipe, etc)

I'm guessing the above isn't an option for you, but you can try drive an understanding that while someone is on call there is no expectation for them to work on anything else. That means subtracting on call head count during project planning

Think of on-call like medical triage. On-call should triage outage (partial/full) level scenarios and respond to alerts, take immediate actions to remedy the situation (restart services, scale up, etc.) and then create follow-on tickets to address root causes that go into the pool of work the entire team works. Like an ER team stabilizing a patient and identifying next steps or sending the patient off to a different team to take time in solving their longer term issue.

The team needs to collectively work project work _and_ opex work coming from on-call. On-call should be a rotation through the team. Runbooks should be created on how to deal with scenarios and iterated on to keep updated.

Project work and opex work are related, if you have a separate team dealing with on-call from project work then there isn't a sense of ownership of the product since its like throwing things over a wall to another team to deal with cleaning up a mess.

Have you looked into SLO/SLA/SLIs?

if this is just a workload vs capacity thing -- where the workload exceeds capacity, is there a way to add some back-pressure to reduce the frequency of on-call issues that your team is faced with?

are you / your team empowered to push back & decline being responsible for certain services that haven't cleared some minimum bar of stability? e.g. "if you want to put it into prod right away, we wont block you deploying it, but you'll be carrying the pager for it"

Without knowing your context, it is hard to give advice, that is ready to be applied. As a manager, you will need to collect and produce data about what is really happening and what is the root cause.

Clear up first what is the charter of your team, what should be in your team's ownership? Do you have to do everything you are doing today? Can you say no to production feature development for some time? Who do you need to convince: your team, your manager or the whole company?

Figure out how to measure / assign value to opex improvements eg you will have only 1-2 on-call issues per week instead of 4-5, and that is savings in engineering time, measurable in reliability (SLA/SLO as mentioned in another comment) - then you will understand how much time it is worth to spend on those fixes and which opex ideas worth pursuing.

Improving the efficiency of your team: are they making the right decisions and taking the right initiatives / tickets?

Argue for headcount and you will have more bandwidth after some time. Or split 2 people off and they should only work on opex improvements. You give administratively priority to these initiatives (if the rest of the team can handle on-call).

Check out the Google SRE Handbook. Still highly relevant today.

Exec level Framework is DORA: https://www.pentalog.com/blog/strategy/dora-metrics-maturity...

For your level: Your team and org size is large enough that you should be able to commit someone half or full-time to focusing on Opex improvements as their sole or primary responsibility. Ask your team, there's likely someone who would actually enjoy focusing on that. If not, advocate for a head count for it.

Edit: Also ensure you have created playbooks for on-call engineers to follow along with a documentation culture that documents the resolutions to most common issues so as those issues arise again they can be easily dealt with by following the playbook.

Note: This is unpopular advice here because most people here don't want to spend their lives bug-fixing, but in reality it's a method that works when you have the right person who wants to do it.

A few things that worked for us:

1. The roster is set weekly. You need at least 4-5 engineers so that you get rostered not more than once per month. Anything more than that and you will get your engineers burned out.

2. There is always a primary and secondary. Secondary gets called up in cases when primary cannot be reached.

3. You are expected to triage the issues that comes during your on-call roster but not expected to work on long term fixes. that is something you have to bring to the team discussion and allocate. No one wants to do too much off maintenance work.

4. Your top priorities to work on should be issues that come up repeatedly and burn your productivity. This could take upto a year. Once things settle down, your engineers should be free enough to work in things that they are interested in.

5. For any cross team collaboration that takes more than a day, the manager should be the point of contact so that your engineers don't get shoulder tapped and get pulled away from things that they are working on.

Hope this helps.

I've been on a lot of oncall lists... 4-5 per week seems extremely high to me. Have you gathered up and classified what the issues were? Are there any patterns or areas of the code that seem to be problematic? Are you actually fixing and getting to the root cause of issues or are they getting worse? It sounds like you don't know the answer because you don't really understand the problem.

If you don't have enough time to run the system and you have to do new feature work one has to give into the other, or you have to hire additional people (but this rarely solves the problem, if anything, it tends to make it worse for a while until the new person figures out their bearings).

One way that is very simple but not easy is to let the on call engineer not do feature work and only work on on-call issues and investigating/fixing on call issues for the period of time they are on-call, and if there isn't anything on fire, let them improve the system. This helps with things like comp-time ("worked all night on the issue, now I have to show up all day tomorrow too???") and letting people actually fix issues rather than just restart services. It also gives agency to the on-call person to help fix the problems, rather than just deal with them.

I don't think you'll find a single framework that addresses everything you're looking for in your last paragraph.

That being said, some advice:

> Clearly define on-call priorities

Sit down with your team, and, if necessary, one or two stakeholders. Create a document and start listing priorities and SLAs during a meeting. The goal isn't actually the doc itself, but when you go through this exercise and solicit feedback, people should raise areas where they disagree and point out things you haven't thought of. The ordering is up to what matters to your team, but most people will tie things to revenue in some way. You can't work on everything, and the groups that complain most loudly aren't necessarily the ones who deserve the most support.

> balancing immediate production needs with Opex improvements

Well, first, are your 'immediate production needs' really immediate? If your entire product is unusable that might be the case, but certain issues, while qualifying as production support, don't need to be prioritized immediately, and can be deferred until enough of them exist at the same time to be worked on together. Otherwise you can start by committing to certain roadmap items and then do as much production support as you have time for. Or vice-versa. A lot of this depends on the stage of your company; more mature companies will naturally prioritize support over a sprint to viability.

> Manage long-term fixes related to past on-call issues without overwhelming current on-call engineers. Create a structured approach that ensures ongoing focus on improving operational experience over time.

Whenever a support task or on-call issue is completed, you should keep track of it by assigning labels or simply listing it in some tracking software. To start off, you might have really broad categories like "customer-facing" and "internal-facing" or something like that. If you find that you're spending 90% of your support time on a particular service or process, that's a good sign that investment in that area could be valuable. Over time, especially as you get a better handle on support, you should make the categories more granular so you can focus more specifically. But not so granular that only one issue per month falls into them or anything like that.

In general, most IT departments operate on a multi-tier service model to keep users from directly annoying your engineers.

1. Call center support desk with documented support issues, and most recent successful resolutions.

2. Junior level technology folks dispatched for basic troubleshooting, documented repair procedures, and testing upper support level solutions

3. Specialists that understand the core systems, process tier 2 bug reports, and feed back repairs/features into the chain

4. Bipedal lab critters involved in research projects... if your are very quiet, you may see them scurry behind the rack-servers back into the shadows.

Managers tend to fail when asking talent to triple/quadruple wield roles at a firm.

No App is going to fix how inexperienced coordinators burn out staff. =3

According to The Phoenix Project [0], if you can form a model of how work flows in, through, and out of your team then you can identify its problems, prioritize them in order of criticality, and form plans for addressing them. The story's premise sounds eerily similar to what you're facing.

At the very least it's a fun read!

[0] https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...

1) Identify on-call issues that aren't engineering issues or for which there's a workaround. Maybe institutional knowledge needs to be aggregated and shared.

2) Automate application monitoring by alerting at thresholds. Tweak alerts until they're correct and resolve items that trigger false positives.

3) If issues are coming from a system someone who is still there designed, they should handle those calls.

4) You mention long-term fixes for on-call issues. First focus on short-term fixes.

5) Set a new expectation that on-call issues are an unexpected exceptions. If they occur, the root cause should be resolved. But see point 4.

6) On-call issues become so rare that there's an ordered list of people to call in the event of an issue. The team informally ensures someone is always available. But if something happens, everyone else who's available is happy to jump on a call to help understand what's going on and if conditions permit, permanently resolve the next business day.

4-5 issues per week can be a lot or a little, all depending on the severity of these issues. Likely most of the them are recurring issues your team sees a few times a month and the root cause hasn't been addressed and needs to be.

Driving down oncall load is all about working smarter, not necessarily harder. 30% of the issues likely need to be fixed by another team. This needs to be identified ASAP and the issues handed off so that they can parallelize the work while your team focuses on the issues you "own".

Setup a weekly rotation for issue triage and mitigation. The engineer oncall should respond to issues, prioritize based on severity, mitigate impact, and create and track Root Cause issues to fix the root cause. These should go into an operational backlog. This is 1 full time headcount on your team (but rotated).

To address the operational backlog, you need to build role expectations with your entire team. It helps if leadership is involved. Everyone needs to understand that in terms of career progression and performance evaluation, operational excellence is one of several role requirements. With these expectations clearly set, review progress with your directs in recurring 1-1s to ensure they are picking up and addressing operational excellence work, driving down the backlog.

> Often, the on-call engineers are pulled into working on production features or long-term fixes from previous issues, leaving little bandwidth for proactive system improvements.

the way my company does it, on-call rotates around the team. The designated oncall person isn't expected to work on anything else

entirely depends on what those 4-5 oncalls are per week.

4-5 Pagerduty Pages is either 1) bad software or 2) mistuned alerts.

4-5 Cross team requests + customer service escalations, <= 1 Page per week is not that bad, and likely can be handled by 1 week rotations with cooperative team to cover 3-4 2hr "breaks" where the person can (workout, be with their kids/spouse, Forest Bathe) would be a decent target.

For me the best experience across >15 yrs experience was at a company that did 2 week sprints. For 1 week you'd be primary, 1 week you'd be secondary, and then for 4 weeks you'd be off rotation. The primary spent 100% of their time being the interrupt handler fixing bugs, cross team requests, customer escalations, and pages, if they ran out of work they focused on tuning alerts or improving stability even further. So you lose 1 member of your team permanently to KTLO. IMO you gain more than you lose by letting the other 5-7ish engineers be fully focused on feature work.

> Often, the on-call engineers are pulled into working on production features or long-term fixes from previous issues, leaving little bandwidth for proactive system improvements.

Have a backbone, tell someone above you "no".

Alert fatigue. Alert fatigue. Alert fatigue. It's the single biggest quality of life thing that you can do to help with the annoyance that is on call. If you know you're in store for the same alert again and again, or perhaps even know that you know you're going to get paged, it's hard to think about anything else. It becomes then a game of normalizing deviance and burnout: "oh, we just ignored that one last time". Ok, why are they alerts then if they can be ignored? It's just going to murder people's spirit after a while.

Someone gets called in the middle of the night? Let them take the morning to recover, no questions asked, better yet, the entire day if it was a particularly hairy issue. This is the time where your mettle as a manager is really tested against your higher-ups. If your people are putting in unscheduled time, you better be ready to cough up something in return.

Figure out what's commonly coming up and root cause those issues so they can finally be put to bed (and your on-call can go back to bed, hah).

Everyone that touches a system gets put on call for that same system. That creates an incentive to make it resilient so they don't have to be roused and so there's less us-vs-them and throwing issues over the wall.

Beyond that, if someone is on call, that's all they should be doing. No deep feature work, they really should be focusing on alerts, what's causing them, how to minimize, triaging and then retro-ing so they're always being pared down.

Lean on your alerting system to tell you the big things: when, why, how often, all that. The idea is you should understand exactly what is happening and why, you can't do much to fix anything if you don't know the why.

Look at your documentation. Can someone that is perhaps less than familiar with a given system easily start to debug things, or do they need to learn the entire thing before they can start fixing? Make sure your documentation is up to date, write runbooks for common issues (better yet, do some sort of automation work to fix those, computers are good at logic like that!), give enough context that being bleary eyed at 3:30am isn't that much of a hindrance. Minimize the chances of having to call in a system's expert to help debug. Everyone should be contributing there (see my fourth line above).

Make sure you are keeping an eye on workload too. You may need to think about increasing the number of people on your team if actual feature work isn't getting done because you're busy fighting fires.

A couple of things I'd suggest:

* Clearly delineate what is on-call work and how many people pay attention to it, and protect the rest of the team from such work. Otherwise, it's too easy for the team at large to fall prey to the on-call toil. That time goes unaccounted and everybody ends up being distracted by recurrent issues, increases siloing, and builds up stress. I wrote about this at large here: https://jmmv.dev/2023/08/costs-exposed-on-call-ticket-handli...

* Set up a fair on-call schedule that minimizes the chances of people having to perform swaps later on while ensuring that everybody is on-call roughly the same amount of time. Having to ask for swaps is stressful, particularly for new / junior folks. E.g. PagerDuty will let you create a round-robin rotation but lacks these "smarter" abilities. I wrote about how this could work here: https://jmmv.dev/2022/01/oncall-scheduling.html

> ... I oversee a team handling a moderate volume of on-call issues (typically 4-5 per week). In addition to managing production incidents, our on-call responsibilities extend to monitoring application and infrastructure alerts.

Being on-call and also responsible for asynchronous alert response is its own, distinct, job. Especially when considering:

> Often, the on-call engineers are pulled into working on production features or long-term fixes from previous issues, leaving little bandwidth for proactive system improvements.

The framework you seek could be:

- hire and train enough support personnel to perform requisite monitoring

- take your development engineers out of the on-call rotation

- treat operations concerns the same as production features, prioritizing accordingly

The last point is key. Any system change, be it functional enhancements, operations related, or otherwise, can be approached with the same vigor and professionalism .

It is just a matter of commitment.

May I recommend this chapter from the Google SRE book: https://sre.google/sre-book/being-on-call/

As well as this two from the management section: https://sre.google/sre-book/dealing-with-interrupts/ and https://sre.google/sre-book/operational-overload/

This sounds like a cliche stereotypical IT problem. And firstly, not a not a bad thing, because it's new to you. Luckily there are mountains of best-practices for addressing this issue. Picking one feather from the big pile, I'd say your situation screams of Problem Management.

https://wiki.en.it-processmaps.com/index.php/Problem_Managem...

Your on-calls folks need a way to be free of the broader problem analysis, and focus on putting out the fires. The folks in problem management will take the steps to prevent problems from ever manifesting.

Once upon a time I was into Problem Management, and one issue that kept coming up was server OS patching where the Linux systems crashed upon reboot, after having applied new kernel, etc. The customers were blaming us, and we were blaming the customer, and round and round it went. Anyhow, the new procedure was some thing like this... any time there was routine maintenance that would result in the machine rebooting (e.g. kernel updates), then the whole system had to be brought down first to prove it was viable for upgrades. Low-and Behold, machines belonging to a certain customer had a tendency to not recover after the pre-reboot. This would stop the upgrade window in it's track, and I would be given a ticket for next day to investigate why the machine was unreliable. Hint... a typical problem was Oracle admins playing god with /etc/fstab, and many other shenanigans. We eventually got that company to a place where the tier-2 on-call folks could have a nice life outside of work.

But I digress...

> Opex ...

Usually that term means "Operational Expenditure", as opposed to "Capex" or Capital Expenditure. It's your terminology, so it's fine, but I'd NOT say those kind of things to anybody publicly. You might get strange looks.

I'd say let one or two of the on-call folks be given a block of a few hours each week to think of ways to kill recurring issue. Let them take turns, and give them concrete incentives to achieve results. Something like $200 bonus per resolved problem. That leads us into the next issue, which is monitoring and logging of the issues. Because if you hired consultants to come-in tomorrow, and you don't even have stats... there's nothing anybody could do.

Good luck

I recently wrote about how NOT to do this: https://pifke.org/posts/middle-manager-rotation/

The best framework has been available for a while:

https://sre.google/sre-book/being-on-call/

https://sre.google/workbook/on-call/