HN Explorer

Ask HN: How do you organize a platform team?

6 points by davideberdin ↗ HN
I am a new manager of a large team (12 SREs) that are taking care of the Kubernetes platform in my company. This team is responsible for the provisioning pipelines (for both baremetal and AWS - no EKS is used), the Kubernetes controllers to integrate with other custom services, the observability stack, etc. The total fleet in use is around 6000 baremetal nodes and 1000 VMs in AWS spread over various DCs and regions. There are over 1500 developers actively using the Kubernetes clusters every day for a total of 2500 applications running in production.

The team spends a lot of time in operations as well as solving compliancy issues, vulnerability patching and customer support. The struggle I'm having is "how to drive focus" and avoid to die of operations. The team is large making the Scrum process ineffective. Every time I try to define teams and to split the people I realise that everything on the platform is so interconnected that the moment I would create 2 or 3 separate teams they would start being on top of each other.

What would you recommend to do?

11 comments

[ 2.7 ms ] story [ 36.3 ms ] thread
asim
I wouldn't split the team just because personally I think it'll weaken the concept of the "platform team". Ideally what you want is people with different priorities/focuses within the team. Some on low level ops, others on high level APIs or services. Identifying who's skills fit what and letting them do that is key.

The other thing is just to split the category of work into 3 things; p1 bug fixes / long term projects / support work. Each week just make a note of time allocation for each based on what's happening (sometimes a p1 fix can take up the whole week). Try to minimise the support burden by creating office hours and defining SLAs for the rest of the company.

Make sure your team is not getting buried in support work. What's going to help them is just being able to filter out what's an immediate priority versus pushing off to tomorrow or the day after. Don't let them get bogged down or pinged constantly. Try to make that request flow async.

And most importantly, give them the time to accomplish tasks they think are most important. They are deep in the trenches and know what's going to be p1 vs not. Trust in their ability to guide the outcomes.

davideberdin
Thanks for the tips. I do appreciate them :)

`Don't let them get bogged down or pinged constantly. Try to make that request flow async.` - This is very true yet so hard. Slack is not helping at all with this issue. I will check with the team about office hours or maybe a triaging system to minimize the Support effort.

asim
If you have a #platform team channel don't let that become the place the rest of the company comes and makes unfiltered requests. That's the channel for your team. It's really important to have that boundary, otherwise they don't have a safe space to communicate and have to resort to private channels. People should be able to see what the platform team discusses but not necessarily just start asking random support questions.

Also, try create a knowledge base for the rest of the company. If there's frequent issues that are already resolved which others could just search a page for or some runbook for things they can do themselves it's best to give them that "self serve" method.

davideberdin
Makes sense -- Knowledge Base is something I should definitely tackle. Thanks :)
jrowley
This sounds challenging! First I’d recommend kanban as opposed to scrum for the team, as the team is so large it is probably hard for them to be really working on a single scrum board effectively.

2nd I’d make sure the team had space to do their operational duties, possibly on a rotation so that other members can focus more on platform development. It can be tempting to try to minimize the operational work, especially when coming from more of a product oriented team. But it is important to develop good process to support the recurring operational labor.

Finally within that team of 12 I’d identify the most senior devs and tech leads and work with them closely on these process changes. Try to understand their problems and ensure they understand the goals coming down from the VPs etc.

Sorry if this is basic / already in play. Good luck!

davideberdin
Thanks for the tips! I appreciate them :)

I've been noodling for a while regarding Kanban. Where do you see the main advantages in using it?

jrowley
I think scrum is great when you have following properties: - clear goals - not a lot of interruptions - tasks / projects are particularly timeline sensitive. - you want to measure output and plan initiatives far into the future

Scrum can also have a fair amount of overhead, all of the rituals, eg pointing tickets, refining the backlog.

Also daily standup / status updates for 12 people can be a lot of time.

When you have support queue, operational responsibilities and regular dev work all going on in the same time, you might find you're spending a lot of time trying to ensure some amount of work gets done in a sprint, while support burden / operations supersedes estimates, which then pushes work to next sprint. So the sprint effectively becomes meaningless, because there is always lots of carry over.

Kanban does require a mature, self motivated team too though, since deadlines aren't as clear necessarily.

stocktech
Splitting teams with interconnected and related work, I do something like https://agilesquads.org/. For each sprint, we'll have a planning cadence where we scope 2 weeks of work e.g. Feature X and Feature Y. Each squad would get assigned a feature and they're able to focus on delivering that feature. The benefit of this over a real "split" is that you can have both "teams" working on the same roadmap/project/feature and/or change how many engineers are working towards a single feature. When you include "squad leads", it's also great for career development and leadership.

On product work, we do an oncall rotation of 1 engineer per week that triages issues and handles prod outages. This may solve for your 'help desk' type work.

davideberdin
Thanks for sharing the link and your advices. I've never thought about squads as they are explained in the website.
danwee
> The total fleet in use is around 6000 baremetal nodes and 1000 VMs in AWS spread over various DCs and regions. There are over 1500 developers actively using the Kubernetes clusters every day for a total of 2500 applications running in production.

Seems like they are quite successful already.

> The struggle I'm having is "how to drive focus" and avoid to die of operations. The team is large making the Scrum process ineffective.

Drive focus to where? As stated above, the platform team seems to have achieved what every platform team other there is struggling to achieve. What does "die of operations" mean?

And above all, who cares about Scrum?

mindcrash
Promote some of the best engineers in the group to staff engineers (or try to recruit them but that could be very costly) and let them work on overarching issues in coop with your existing devops and engineering teams.

To grasp staff engineering as a manager (or a potential staff engineer) this is probably the best (and only?) resource around => https://staffeng.com/