Ask HN: How do you balance support and sprint tickets?
Our DevOps/SRE team has been trying to do Scrum for over a year but the sprints are always derailed by support work.
They've tried to add support tickets to the sprint as they come in, but that changes the sprint scope and renders the "velocity" metrics useless.
They've tried to exclude support tickets from the sprint and keep the scope very small. Some weeks it works, other weeks they overdeliver by a lot (because there were fewer support tickets and they got a lot of tickets from the backlog).
I'm thinking Scrum is useless for that team but management insists all engineering teams must use scrum.
Do you have any advice for managing tasks in a team that has support/adhoc and project tickets?
121 comments
[ 5.0 ms ] story [ 205 ms ] thread* The team does short sprints.
* One lucky soul handles all support requests for the duration of the sprint, having no officially planned work for the sprint.
The staff take it in turns to be the "support-person", and if they have a lot of tickets they work a lot. If they have few then they might sneakily do some work, or help colleagues with their own stories/tickets.
How short is short?
Just do the most valuable thing first. Ship. Repeat.
Orrrr.....you assign points to the support work as well, your team hits their 50 points every week or else they get hit over the head by management as to why they weren't on track.
This also helps avoiding constant interruptions or context switching to the whole team that are now handled by the designated person of the week.
As an added bonus, this incentivizes better documentation of your system via runbooks (I'd rather point someone at a doc than walk them through a problem when it's not my week).
And eventually employees burn out and start mailing in that week and now suddenly nobody cares except the other teams having trouble getting support, but apparently that is a diffuse enough complaint that nothing ever came of it.
Not that I speak from experience...
That works out to leaving one team member a full sprint for a deeper heads-down task.
To alleviate some of the extra load on the on-call rotation, we dedicated a decent portion of our sprints to automation for a few quarters, which allowed us to reduce some of the recurring manual tasks and ease the load on the on-call role. Now the on-call person will usually see an average 15hrs of support on the regular (of a 40hr week), down from damn near 35hrs.
That is to say there was about 20hrs of automatable tasks that had been shouldered by one team member for years. When that person left the tasks were placed onto the on-call rotation as a duties as a way to ensure every team member became familiar with these tasks instead of being siloed into a single person (there was a scramble to learn those tasks as this person was offboarded, and a rough couple of months as we skilled up on these).
We do it similarly. We generally reserve 20% of our time for production support and 80% of time for new development. If we have fewer production issues that week, we get more new development done. If we have more production issues that week, get get less new development done.
We have one generic production issue story where we put small things and we'll create a fully fleshed out story for larger production issues.
If we have a trend of having too many production issues then we raise that as an issue itself and get to the bottom of what is causing it.
All 'support' created tickets go into the backlog then, earliest they can possibly be addressed is next sprint. (Assuming you want the rule to change/it be someone else's problem/proposed solution.)
As a bit of an aside, I'm a bit curious about your role/the org structure that you seem to have oversight over SRE as well as other teams' working processes, but also have 'management' imposing roughly what they look like from above?
If there is any one thing teams new to Agile need to learn, it is that Scrum is just one choice among many. And nowhere near as universally useful as people seem to believe.
What we used to do (while growing the SRE team from ~8 to 30) was having dedicated teams on specific projects. Each had their own scheduling system as they saw fit. AND a weekly rotating support duty which took precedence: teams had to allocate someone from their own capacity to this duty.
The importance was to adequately rotate everyone, to pair a senior and a junior in the support team, and to be available to them to debrief, or to reallocate high priority/criticity tasks to more expert members, if required.
Also, if the support queue was empty, they could do maintenance/exploratory work, but NOT their own team project work.
HOWEVER; The counterpoint would be if you are doing new product work and your priorities are changing faster than a week or two you have either an exceptional situation or this should be a canary that you have something really wrong with your priority directions.
In the latter case scrum is being a useful tool in bringing this to your attention.
1. Pre-customer development
2. Contract development shops where you bill by the sprint
I’ve never seen good outcomes in a live product environment.
Usually the answer is Kanban for smaller teams or Scaled Agile for larger teams IMO…but the situation the OP is dealing with looks like they are prioritizing preventative work to stop these support tickets from having to come in in the first place.
Prevention measures for unplanned work are critical.
I STRONGLY disagree with this, conceptually, logically and in practice. Scaled agile approaches make you plan your sprints way further in advance and account for emerging (or sustainment) work with the use of reserving a percentage of capacity. This works about as well as riding out a run of bad luck at the casino while you wait for the odds to balance out. Scaling "up" the existing practices to a department or company level will not help an interrupt-driven team. Scaled agile doesn't work for pure development teams; i'ts even less likely to work for an SRE team.
It gets the plan in front of everybody so you can discuss the tradeoffs and risks too. There's an entire open session as part of it called R.O.A.M. where risks to the ability to deliver the plan are discussed, not just with the tech people but with the business folks too.
That portion of PI planning is where you call out things that could break the plan. Emerging (or sustainment) work is one of those risks and you have to call it out, then once it's called out you have to discuss what the organization is going to do to prevent it. Capacity isn't going to just spring forth out of nowhere, so what are we pushing off if there's more of it than expected? Do we need to prioritize preventative work to make sure we don't have problems? Do we need some type of infrastructure, monitoring system or automation in our devops process?
Everywhere I've been, the PI Planning process has led to less "we have to do this now" and more "we need to see if this can be included in the next PI", which creates significantly less disruptions to the development team focus. The only remaining interruptions should be production issues (preventable) or significant market shifts (rare).
It doesn't work when people try to adhere to that plan so rigidly that it becomes a waterfall system.
IMO the biggest issue with Scaled Agile always comes down to leadership IMO. It's not supposed to be rigid. It's supposed to be adapted to your organization and put the plans in the hands of the developers.
The horror stories about SAFe that I've seen on here usually end up with me asking a few followup questions and those questions make it clear that something was terribly wrong. One guy on here told me they did PI Planning for 2 full weeks, which is abject insanity.
> management insists all engineering teams must use scrum.
They aren't doing scrum. At the very least, just "do scrum" externally and internally do kanban. Management is clearly dumb enough to believe it.
I have a blog post on the subject a while back: https://blog.rstankov.com/bug-duty-process/
We have one "on duty" person each week/half-week/two-weeks. Their priorities are:
1. Support tickets.
2. Whatever they want (usually code cleanup).
They aren't expected to get any project work done. We also track the amount of time that they are spending on support tickets. If it is getting high we move more cleanup tasks into the regular project work cycle.
If you have oncall you can often co-locate these as well (oncall is priority 0).
The main risk is that if your support load gets high than there is no room for cleanup/automation which can become a feedback loop causing more support load. I haven't seen a good solution for this other than keeping track of the load and scheduling proper "project work" to address it when it gets too high.
This is part of the process. If an issue requires deeper work, it moves a proper task in a sprint.
We try to fix the easiest issues first so the deeper problems bubble up.
How the person in the support role knows that an issue is deeper? It comes from experience. Thus everyone does 1~2 week of support.
Scrum is an accounting tool. Apply all the shenanigans that one might find in financial reports to it.
At my first job, we just edited estimates after the fact to hit the velocity target management wanted.
OK, this is a problem because why? Some weeks it works, and other weeks they accomplish more than expected... and this is a problem why??
Like, what's the actual problem here, if good progress is being made on features and bugfixes?
My guess it's a problem if management is trying to use "scrum" to treat developers like sweatshop workers who they wring the most possible production out of. How can we know what the most possible production is, if they have a different amount of time to work on it every week?!?
We could come up with different arithmetic ways to solve this. Assign "points" to support tickets too (even after the fact, how much time they actually took?) to include them in your "velocity". Or keep track of how much time is being spent on support to, to "normalize" the velocity of the other stuff accordingly (different way of doing the same thing).
But personally I do not really want to help companies become better at treating developers like sweatshop workers to wring maximal productivity out of. I'd rather realign to "agile" it was originally intended, to empower developers to bring value, not to treat them as commodified interchangeable widgets to exploit and burn out.
but, I mean, exploitive companies gonna company I guess.
> I'm thinking Scrum is useless for that team but management insists all engineering teams must use scrum.
Which they also insist means relying on those "velocity" metrics? Can you do "scrum" without "velocity" at all?
OP didn't mention it but it doesn't take much effort to imagine many weeks will have lower velocity because of influx of support tickets. Management focused on metrics won't be much happy.
Rotate who the "fireman" is on the team, they are out of Sprint and just fix. Spreads internal knowledge around too.
One thing to consider… support tanking your velocity is a feature, not a bug.
Sounds like you need investment in support tooling etc that can lessen that burden. Velocity will improve when support is less burdensome.
This is obviously not one-size-fits-all advice but worked well for a previous org I was at. If the sprint work will have a side effect of fixing support or support is so intense it requires a majority of the team you HAVE to fix that first regardless if sprint/scrum/kanban, etc. if velocity was super high for a few sprints but regressions/bugs were introduced they could impact future velocity, so those high velocity sprints weren’t as productive as you thought. It’s never just one number.
The org I worked at with the absolute worst support/sprint structure also had the only code base I considered “unsalvageable”. They refused to do anything to improve existing processes and spent half of every team’s time on the same support issues on an endless loop. They never had the measurements needed to actually figure out where to improve clients experiences.
If that many tickets are coming in and disrupting everything, I’d be using that to prioritize addressing the causes so the disruptions stop.
If you’ve ever read The Phoenix Project, this is well illustrated when they talk about prioritizing preventative measures for “unplanned work” and they map out in detail why it’s such a huge problem.
I have never worked anywhere, including using scrum, where I could avoid incoming issues for 2 weeks at a time.
At the end of the day, better systems, better code, less technical debt means less support. Issues with previous work might indicate a false velocity of the past, releasing low quality.
There are long term maintenance exceptions that happen to all code bases, that have nothing to do with quality, like supporting new platforms or unpredictable platform changes, those can go in as backlog tasks.
The real question is what you are using "velocity" for? Is it to help get a feel for work you can complete, or is it a metric to be judged by management to evaluate you?
It should, but I've been surprised by some companies interpretation of "product velocity" to just mean "how many tickets does each developer complete per 'sprint'".
That's not product velocity, that's just some micromanagement nonsense so they can measure/push devs. No wonder many devs hate "agile" like this.