Ask HN: How “normal” is for a developer to be on-call?
A few years ago, my managers decided that developers should be on call for incidents. We didn't have any sort of first responder or site reliability engineering staff, so almost all incidents, no matter how small, would be handled by developers.
Things improved a little bit after some time, but the idea of on-call developers is still very much ingrained in the incident management process.
Now, personally, I don't quite understand why a developer would need to be woken up at midnight to diagnose something that cannot be patched by him or her alone.
Perhaps the issue is that our process is dysfunctional, but I'm wondering if it is necessary, or just common across the industry.
20 comments
[ 2.7 ms ] story [ 37.4 ms ] threadWe had an informal rotation in the dev team so there was always a named person who'd make sure any customer issue got an intelligent first response - up until a certain point in the evening. That was to ensure we were able to ask for data we needed from customers in other time zones, then look properly when the team were in the next day.
I have known people on 24-hour call in companies that had infrastructure that was critical to their business (i.e. "as a service" sort of deals).
(personally, where I see developers are already doing some out-of-hours support work voluntarily, I think overformalising would put people off and be counterproductive)
truly being a team player would be to realise the team identity does not include the company but is instead built around professional identity as fellow bit-gardening day labourers who exchange their time and energy providing engineering and operations services to the owners of the company in exchange for money -- then banding together collectively (perhaps starting a union?) to increase negotiating power to obtain better pay and working conditions
if anyone has a particularly pithy response to "you build it, you run it" along the lines of "if you'll pay both of my invoices for performing both jobs, OK", i'd love to hear it
This, plus "you build it, you run it", which I don't agree with, given that most mid to large sized companies run devops teams, effectively hiding infrastructure details from developers.
Isn't it also true that a developer who routinely misses sleep, is less productive and makes more mistakes during working hours?
But I have heard your argument before, and it kind of rings hollow. Unless you are willing to allow developers to push changes to production in the middle of the night and without supervision, what exactly is the benefit of having them on-call? Saving money?
Your point is correct thought that the responder does need the ability to make a hot fix. Non actionable alerts are of negative value to the company.
You should read The Phoenix Project, that will help you understand the rationale.
What makes you think I haven't read it? When my company went through its so called "devops transformation", first thing our devops advocate did was giving away a dozen or so copies.
I understand the rationale. I just happen to not agree with it.
In my mind, it's naive to believe that a team would have more than two or three competent coders, with a thorough knowledge of the code base, the high level vision of where that code fits and interacts with the rest of the infrastructure, the skills to quickly fix things, and, on top of that, the willing to be on-call several days a month.
And then there is the fact that nobody looks for solutions at 2am, just workarounds.
How many at a time? It has to be two or more, otherwise anyone could push changes to production at any time, without supervision.
My partner is an executive chef at a private school. It's one of our country's top schools that gets the occasional visit from people with royal titles (and one might soon be King), so think super rich kids: there is a lot of pampering. All of a sudden through this Covid thing we're all working through, there are texts and phone calls at 5:AM "There's a kid with a new allergy coming in today. Make changes in the lunch menu to accommodate them." This is literally a situation where one could ask, "Shouldn't this be an email, CC'd to others involved, too? And maybe even the day before so appropriate food items can be ordered?"
It's super annoying and clearly goes against how things are normally done in a well managed business. A lot of companies are clearly taking advantage of all the workarounds this pandemic has forced us to contend with. Except usually the abuse isn't even related to pandemic conditions.
A lot of people are technically working 24x7 nowadays.
Funnily enough, we went through something similar during our “agile transformation”.