Being on call is the bane of my existence. It’s also entangled with the issue that companies would never consider people working a night shift even tho tons of devs prefer to code at night (and they could actually get things done without constant context switching). How is it that we are expected to be on slack, checking emails, and on call after hours but employers rarely pay for cell phones or after hours labor?
"I don't have a phone number, it's data only and my house doesn't have good reception" is what a number of ex-colleagues used when asked to be on-call at companies that asked for it, but gave you nothing for it.
For companies that paid you overtime, it often starts with guilt, "well the rest of the team did it last week/month/year, it's your turn". If they don't incentivize you enough to do it, don't do it.
I manage a team which operates services (among other things) for clients. Our aversion to being on-call drove us to build robust systems, automate the heck out of everything and monitor as much as possible. That allows us to spot issues during the day shift before they become problems for the night shift, so on-call duty became over time a relatively relaxed affair for the team.
My team has what we call "the strike team" which is not just on-call but even during the day, your job is basically to make everything more robust (as opposed to what we do normally, which is work on new systems and features). So just last week or so there was an alert on Sunday that I then spent the week to fix permanently. These are also services that my team are the sole developers on so I know when I fix something it will generally stay fixed.
On top of this, we have a rotation so each of us is only on-call one week out of maybe every four or five. And although I agreed to it mostly because it was a condition of the job and I wanted the job more for learning how the team worked than I cared about the money, the compensation for being on-call is actually pretty good even if nothing happens. And if on-call lands on a holiday, we get the holiday time as vacation days to spend later.
So overall, while I would prefer not to be on-call, I feel like our team implements it about as well as can reasonably be expected. I expected it to drive me crazy, but it actually hasn't yet.
I used to work for a small systems integrator. We used on-call mobile phones, the "hot potato" was carried by the on-call engineer. Any actual time worked outside of core-working hours was paid back in the form of time-off-in-lieu. The salary generously reflected the being on-call requirement.
The other factor was that a "call-out" was not completed until the root cause was fixed.
I believe the real reason for the compassionate arrangements was that the owners of the business were former engineers and were even available to escalate calls to them if you got stuck. Our personal phones had everybody else's personal numbers in the address book, but we were never permitted to give them out to clients. Clients only had access to the "hot potato" phone numbers, which also received the various paged alerts, etc.
2. There are few pages, preferably the median should be 0 per week.
3. Spurious / non-actionable alerts get fixed right away (with very high priority)
4. You're not up more than 1 week per 1-1.5 month.
5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime.
As with most things, the core idea is not bad, it's the execution that matters.
If it's in the contract you sign, then it's already priced in. I don't see much value in specifically outlining which part is base and which is for oncall, if oncall is mandatory.
Surcharge for on-call activity is an incentive for the organisation to minimise on-call. If I ask my boss for resources to automate operations, he's going to measure that against the cost caused by on-call activity. If there's a flat-rate cost for on-call, the organisation has one incentive less to improve operations and reduce incident count.
Because if you are supposed to be prepared for sudden work outside your normal work hours, you should be compensated outside of your standard pay. Period.
And sure, it is priced in because the companies can get away with it. This doesn't mean it is right or fair as the price is always going to be in the company's favor and rarely fair to the employee. For example: My brother worked for a US railroad. He didn't have a set schedule. Instead, he got 10 hours rest after a shift and then he was on call. They only closed down on Christmas and New Year's. You were expected to do this on-call work perpetually. The money was good for the area as were the benefits. They advertised in depressed areas without much opportunity, so it made it easier to prey on folks that will accept the poor treatment. I'm pretty sure having to pay folks for each of those hours would change the behavior of the company in the employee's favor.
So yeah, even one hour of 'on-call' should be paid extra, outside of your basepay. Even for companies and industries that aren't being actively evil so that it doesn't happen in the future.
My contract says that if on-calls are needed, I might have to be in the rotation. This clause increases my pay rate even if I'm not on-call.
If however I am actually on-call, I am paid more. And if the on-call rings, I'm again paid more on top of the on-call period. And as the French law mandates 11 consecutive hours of rest, if the on-call rings in the middle of the night, I'll usually come to work later the day after.
If you don't have advantages for being on-call, you're the one being taken advantage of.
In a US state with at-will employment, on call can be added to an existing contract without additional pay - continued employment is considered to be sufficient "consideration" on the company's part.
Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available.
Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work laptop.
It means never going to the beach alone on a Saturday because while you could bring your laptop with you, if you go swimming and there's an on-call event you can't address it because you have no way to get notified while you're swimming in the ocean.
It also means going to the movies with an expectation that if you get called 15 minutes into the movie you're leaving. Likewise, if you're mid-date and get called you're out of luck.
It means never enjoying being able to walk around while being disconnected from the world. It means if you're at your mom's funeral giving a eulogy you leave mid-speech to address PagerDuty.
Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second.
> Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second.
And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep.
It's happened to me, I felt awfully guilty for not waking up after a gruelling work day and some pages early in the evening. I was tired and had been asleep for around 2 hours, didn't wake up and the escalation policy took it up to my manager... I didn't get reprimanded or had any bad consequence from it, still the guilt made me feel like a failure and increased my anxiety when I'm on-call.
>still the guilt made me feel like a failure and increased my anxiety when I'm on-call.
I don't know if this will help with your mindset at all, but as someone who's been listed as the secondary if the first responder doesn't answer: I really wouldn't beat yourself up over it. Self-correction is fine, but try not to let it drag you down too far. We have the same worries too. Our worries are usually "Crud. If I don't pick up, then it's either going to (big boss) or the customer is going to be firing off nasty emails in the morning and dragging me into some stupid meeting." The cause of why we're being paged doesn't really come into play. At most, I might just send a text or whatever to the first responder, to make sure that they're okay. "What if they aren't responding because they were in a car accident?"
If you have a good manager - and it sounds like you do, since you weren't reprimanded for performing a fundemental human function - then they are probably doing their best to look out for you and your interests. When people talk about "not being the boss" or "working as a team" or whatever, generally what they're trying to get at is that "They have your back". They've been in your position and have probably felt that same crush of emotion and worry that you might deal with. They are meant to be the final filter, in a hierarchy of filters, to protect you from those outside elements which ruin your ability to work effectively.
If you put in a sincere effort at your job, you have nothing to feel guilty over. That might be easier said than done, but it really is true. It just might take some time and practice to be able to forgive yourself when you make a mistake.
And if you feel like you could be doing a better job at this, or at that, then you just need to permit yourself the time to improve on that thing, and to remind yourself that you are allowed to mess up from time to time. It's about how you learn from those things.
As someone who is second/third-tier on-call for most of the year: You know what's worse than getting paged when first-tier doesn't handle it? Not getting paged.
The only person I ever had to formally reprimand for on-call policy wasn't the one who failed to answer on time, it was the person who, three times, acked and didn't say or do anything else only because they didn't know how to fix it. In all cases I ended up getting pinged by someone else (head of our support team, or a colleague working an odd schedule), after the situation significantly worsened.
Higher tiers are there for a reason; don't feel bad if you've used them, only if you're expecting to use them.
(On the other hand, feeling guilty for making a mistake is also human, and to some degree something that makes me want such a person on my on-call team. People who feel better when they do a better job will do a better job!)
I would humbly suggest that if this person didn’t do anything, either they needed to be better informed of expectations or that your culture needs to change so that they don’t feel the need to ack without actually doing anything.
+1. I am a deep sleeper and Ive missed pages at times that got escalated to the secondary. Nothing major, just something that autoresolved. But the guilt I felt was… incredibly stressful. Because I felt I was letting down people I was familiar with.
You're using "never" everywhere here. That is in my opinion the main red flag here.
On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds having the pager in such circumstances if it almost never pages outside of working hours.
The company having that much influence over your day-to-day life during non-business hours every fourth week is an enormous burden. I would want at least 50% higher overall total annual comp to even consider that schedule, and even then it's still only a maybe.
almost always there is somebody who's plan is "sitting at home" and nobody minds having the pager in such circumstances
I wonder if this is an American attitude that exists primarily because we (collectively) have allowed our employers to demand this of us?
Personally, I hate being on call. Most weekends, I spend at least 6 hours cycling. Sometimes significantly more. I go camping regularly. I go kayaking regularly. If I don't spend time outdoors, my mental health declines (really, ask my wife, she kicks me out if I sit around the house too long because I turn into a cranky butthead). Most evenings after work, I run/cycle/hike. I have to run errands. Walk the dog (usually 2 miles). This is all part of my normal "not doing anything special" time at home. None of which is easily doable if I'm on call.
Fortunately, I've managed to build a career in a place where on-call rarely exists.
I believe the point was more than if you have a team of, say, ten people, there's always someone who is not busy for some given night, and there's some reasonable trade that can be worked out, the moreso if oncall risk is considered low by the team. Obviously as you scale down that becomes less true.
IMHO the whole notion of "week of on-call" is ridiculous. Being on-call ready is a shift of (hopefully low-intensity) work time. You can do 8 hour shifts, 12 hour shifts, 24 hour shifts, but you can't do 168 hour shifts for monitoring something, anything - that's impractical and should be illegal.
There are (or should be) some minimum standards of rest time that every human being must get, and being on-call is not it, every employee must get an opportunity to fully disconnect for non-trivial time during every single week. You do your shift, and then get at least 12 hours off (with your phone off) before being available for work or work calls again.
On-call is meant to be for rare emergencies. If you're getting paged outside of work hours on each 1 week shift then something is wrong. It should in no way be equivalent to 168 hours of work.
If there is an obligation to respond within a certain time, and an obligation to avoid certain activities (whatever you imagine that makes you unable to respond) then those are work hours even if nobody calls. It's exactly like a fireighter shift or call center shift where there happen to be no incoming calls, or a night shift at a remote gas station if no customers come during that night.
And if you're getting paged outside of work hours, then there's zero obligation to answer your phone.
There's no middle ground. If the employer says that these aren't working hours, they have zero right to ask what you're doing at these hours, much less put any conditions on it; it's your right to spend that time fishing in a remote lake with no cell phone service or go on a date or whatever and not even explain anything without any reprimand when you arrive at your scheduled start of work time; and if they want you to commit to a shift, well, "a shift" where employer tells you what to do (e.g. do not go fishing to a remote lake and don't sleep for 12 hours with your phone off) is by definition work hours.
It is quite plausible that most of the times most employees will pick up the phone and solve reasonable issues, but that's an extra courtesy from the employee, going beyond what you can demand or expect; but the moment you start to require that, or ask an employee to precommit that they will definitely be monitoring their phone for rapid response, that means you're effectively assigning those hours as work hours.
It's understandable that being on-call can be very light work in many cases (not all - quite a few counterexamples in this discussion), so you can agree on different compensation for them, but those definitely are work hours (they definitely aren't non-work hours, and there is no middle ground) and thus any rules on length of shifts and rest between shifts can and should apply also for on-call hours.
If someone works an on-call rotation and was told about it during the interview process then I see no issue with it. It's baked into the agreed-upon compensation already.
I don't see how you can declare what an employer/employee are allowed to agree to. Working an on-call is completely reasonable for a salaried employee. If the terms changed after you were hired and you're bound by a contract (and cannot leave without penalty for a certain timeframe) then things would be very different. But the majority of the HN audience is in the U.S. where employment is generally at will.
> I don't see how you can declare what an employer/employee are allowed to agree to.
Okay, I am coming from a non-US perspective where it's obvious that you can declare what employer/employee are allowed to agree to - "employee rights" means those things which are unalienable and nonnegotiable.
I'll simply quote the universal declaration of human rights "24. Everyone has the right to rest and leisure, including reasonable limitation of working hours and periodic holidays with pay." - not only do countries have the right to intervene in employer/employee contracts, they have a duty to do so. If local employment law permits companies to require salaried employees to work 168 hours a week, then that law is literally enabling violation of human rights and should be changed.
and that's great in theory, yet incredibly vague. I doubt it's targeting HCI that are predominantly on-call: medical professionals, infrastructure engineers, programmers, etc. I picture it as critical of 996 working hours and similar.
In any case I suppose we have different expectations on what's reasonable. If you think a handful of pages per year from my consentual employer is infringing on my fundamental human rights then there's not much I can say to convince you otherwise.
I'm looking at this exactly from the perspective of jobs like medical professionals, infrastructure engineers - I have worked in power infrastructure and have relatives in medicine - where locally none of them are on-call (because the law proscribes this clear boundary); they do 12-18-24 hour shifts of work time, and once the shift ends, they can be (and often are) unreachable. It's not vague, it's extremely clear and simple. Even in a low-volume alert-monitoring position for infrastructure where you might get a few incidents per week (so zero incidents on a median shift) every shift is work time, and scheduled appropriately.
It's not about the number of pages you get, but about the differing expectations on boundaries between your employer and your private time. I do have a strong expectation that employers should not be able to set any restrictions of what people will do in their non-work time, or even ask whether the employer will be out-of-service this weekend, in my opinion that is crossing a line.
As I said, it's entirely reasonable if you get a handful of pages per year and service them - I also have been in roles where I got a handful of night calls per year and happily did what was useful, and I'd expect that in most cases that's okay for most people. However, if it becomes a requirement where I or you must be "on alert" for a whole week, then IMHO that's not reasonable anymore; I answered those calls but I had no duty to be ready for them and abstain from activities that make me unreachable, and if I had a contractual obligation to do so that would be unreasonable (and also a void, unenforceable clause violating employment law). And if you did not volunteer for it but an employer "pushed" this unreasonable requirement on you (e.g. as a condition for employment) then I would actually say that this violated your human rights even if you got zero pages per year; employees should have the practical right to freely decide how they spend their non-work time during the week (and they should have appropriate non-work time) no matter what contracts they sign.
> where locally none of them are on-call (because the law proscribes this clear boundary); they do 12-18-24 hour shifts of work time
24 hour shifts? Sounds like a real worker's paradise. I'll keep my 7-8 hour days with bi-monthly on-call over that, thanks. I'm glad you aren't legislating where I live; I am much happier with the choices and trade-offs in my life than what you or any euro-crat could dream up to "protect" me.
> if you did not volunteer for it but an employer "pushed" this unreasonable requirement on you (e.g. as a condition for employment)
You have a very different concept of what voluntary and self-agency mean than I do.
I've definitely seen on-call compensation implemented differently than stated in the collective agreement. And not in the positive direction. Thankfully it wasn't mandatory, so it was a very definite possibility to not opt in.
having resonable response times make a huge difference in being oncall.
in my experience, most employer seem to think responding within 30minutes or so is doable, and i tend to agree with them.
Usually makes it possible for you to just do things around the house/ in your life, and not having to rush back home when you run into an oncall issue.
In our case it's one hour, essentially based on "what if my home internet doesn't work and I have to get to the actual office to start responding, how long will it take me to get there from anywhere in the city." You don't go on vacation or plan your wedding, but otherwise your day is pretty normal.
At one place I worked, they were wanting on-call shifts of 12 hours being no more than 15 minutes away from being logged on (it was a multi-week Big Event) with the promise of, maybe, time off in lieu as recompense.
They were most displeased when I declined this opportunity.
Also, only on on-call for things you're actually responsible for. I hate having on-calls span multiple teams. They can then be lazy, as their issues hurts someone else. And the added stress of having to debug and fix shit you're not comfortable touching.
Additionally, I'll never accept on-call with 15 minutes from alert to being on a computer again. It's just too limiting and disruptive of my life. Have to bring the computer everywhere. Any dinner or social event can be instantly ruined. A workout becomes meaningless. I remember doing a swim and having to check my phone every 5 minutes. It's mentally exhausting and frankly not worth the pay.
I would add a "there is time reserved to write automation to reduce toil".
I have a fair bit of experience in teams with developers hating oncall and the critical issues happen in two camps generally:
- In some cases the org was absolutely open to give them time to resolve issues and automate stuff away (PMs were proposing months for fixes only, and cleaning up boards, etc), there just was no interest until enough escalation happened (creating a massive conflict between ops and devs in the meantime). Even offering to do the work was met with a "stay in your lane" kind of response and no collaboration at all.
- In others the company just did not care, deprioritized tickets until things blew up completely, then finger pointing started and all that nice toxic bullshit.
On-call means you have to plan your free time around being available to work. That's never going to be "just fine" for some of us, no matter how it's structured.
> 4. You're not up more than 1 week per 1-1.5 month.
That seems excessive if you're expected to be able to log into your work system within X minutes.
Having to be essentially home, near a computer, 25% of the time (1 week out of 4) is a pretty heavy burden, especially for people who prefer to be out, rather than home.
It's a heavy burden, and a lot of teams might want to consider a longer interval, but there's a lot of legitimate scenarios where it's just not feasible to distribute an oncall rotation among 8+ people. I would definitely point more towards 1 in 6 as the ideal minimum.
Yeah oncall is horrible. If things needed to be up 24/7, then some team should be staffed 24/7 around the world.
The worse part of oncall is the control of your life it has. for one week I can't do anything I would normally do. (if your company actually compensates for this, let me know where i can a apply, or better, if it doesn't have oncall at all!) Of course managers are never oncall 24/7. The worse is they give the excuse well im on call all the time by default since im the one manager. But theyre not reorganizing their life and putting their off work hobbies on hold becasue of it are they?
> a monitoring change that fixes some flaky alert that might page somebody about once every six weeks.
These kind of things suck. I was on a team where we had tons of these, 10 alerts like this mean your getting pages all the time. No single alert is worth the time investment. Worse was a manager insisted there will always be a base line of alerts that go off and we will just live with it.
Teams never seem to understand how to alert on stuff. Ive been paged for things going off, that might indicate a problem, then you get stuck sticking around because someone else wants to just wait and see what happens. "We should just be cautious" Its impossible to push back on these things, your just going against someones gut feeling, like maybe one day we will want to know, and everyone needs to protect them selves.
The problem with oncall teams that are different to the usual SRE/DevOps teams are the lack of understanding of the system. This can obviously be fixed with good documentation, but in reality, no one has good enough docs. The second problem is actually building that team with the skill needed. Someone with the skill to fix complex systems is not going to want to stick around as 1st line on call support.
yeah thats true, there must be some kind of middle ground between an ops team like that and the can't go to the bathroom without your phone oncall we have though.
Most places at some scale have some meaningfully defined escalation path. That way you staff people with varying understanding of the system(s).
> This can obviously be fixed with good documentation, but in reality, no one has good enough docs.
One problem I've witnessed related to documentation about on-call issues is the over reliance on the SOP concept. They only commit to one level or one pass of analyzing the issue. They do not future drill down, either by linking to other notes or reviewing the issues deliberately. It's like they read about the 5 Whys and decided why not just 1 why.
> Teams never seem to understand how to alert on stuff. Ive been paged for things going off, that might indicate a problem, then you get stuck sticking around because someone else wants to just wait and see what happens. "We should just be cautious" Its impossible to push back on these things, your just going against someones gut feeling, like maybe one day we will want to know, and everyone needs to protect them selves.
From an ops person: if an alert does not have:
- clear, provable impact on customers (internal/external)
- clear documentation (e.g. runbooks) on how to solve it
It should not be an alert. I took this path (successfully) when trying to remove spurious alerts that existed only for the ego of someone (most absurd example, something that started complaining when p99 for some endpoints went >500ms and happened every day when we downscaled the ASGs because business hours were over. No clear path to resolution, and impact was a couple pages opened a bit slower sure - but the number of customers using those pages after hours was <1%!
It sucks, definitely, but the best way to go around those alerts is to prove they're pointless or a waste of time or can be automated around and should automated around (and I've seen so many servlets leaking memory triggering OS alerts for OS teams or spawning infinite threads and never cleaning up after themselves...).
If the company does not want to do it, and pushes back, I would recommend starting to look for another company. It's sad, but it is what it is. 99.99% of software does not need a follow the sun rotation (or people damned to night shifts), just a bit of thought about what happens when things fail.
There are many issues with being on-call, particularly in environments where false alerts routinely happen, and where management aren't in the rota so don't directly feel the pain. One of my biggest though is the concept of a weekly rota, with people being on-call for a full week at a time.
Sometimes that works fine, and you'll get no alerts all week, but incidents tend to cluster. If something has changed that caused an incident odds are it's going to have knock on effects, and you'll see more alarms over the course of a week. With a weekly rota you end up with one person handling that, who by the end of the week is completely destroyed.
Anywhere I've been responsible for setting up an on-call rota I've instead gone for daily rotations. That means if you were up in the night last night, someone else is going to be in the night tonight. It also means if nothing happens you don't have to spend an entire week either cancelling plans or lugging a laptop around with you just in case.
Daily rotation seems too frequent to me. Sometimes you need 2-3 days to really deep dive and fix the core issue after the alarm has gone silent. Also, with daily rotation it is tempting to wait it out and punt the problem to the next person.
Fixing the core issue should be done during regular day hours on a normal work schedule, so it does not justify a longer on-call shift. If something needs 2-3 days of deep dive, you should get proper time for rest (i.e. not on-call) between those days.
Yeah I have a vivid memory of hysterically crying after getting less than 12 hours of sleep over 3 or 4 days during a particularly bad shift. My pager went off 30 minutes before the end of my shift and I lost it. I should’ve passed the pager but I was too “up” to recognize it and my manager at the time was completely clueless about oncall work. Actually at that same company getting paged 8-10 times off-hours during an oncall shift was not unheard of, the burnout was real… I’m surprised there aren’t more lawsuits regarding this because the toll of sleep deprivation on health is well known.
Thanks for sharing that. I had a similar experience at my last job. The emotional/physical toll is so real. On the one hand, guess that's why we get paid? On the other hand, screw that.
How often does someone get a daily rotation though? One week out of four is not uncommon. One out of every four days on-call would really destroy your life though.
My company(UK) recently tried to force on-call on all engineers.
The initial wording was very restrictive, like 5 minute acknowledgement time and 15 minutes at-laptop. 24/7 for 7 days. They tried to have this implemented without any extra remuneration or perks for the on-call engineer.
On top of it possibly being very illegal, it seems very immoral to spring something like that on people that did on agree to it when they took the job.
I fought for it and I got them to change their policy in 2 mostly meaningful ways:
- It's an opt-in method
- On-call engineers get paid extra for just being on-call and get extra time off whenever they need to actually do something.
This makes sure that you only get people actually willing to do it and there is an incentive. I think it's been quite a successful program!
Luckily I didn't need to get them involved, but in the UK there are unions starting to form for tech workers, I suggest you join one like https://prospect.org.uk/tech-workers
A company I used to work for asked me to do on-call, it wasn't in my contract, I declined, that was that.
I don't understand what "force" means in this context - the conversation went something like "I have commitments outside of work" and that was that. I mean, there was a back and forth, but yeah, at the end of the day I took the job knowing I'd be available for the hours they wanted when I took the job.
Indeed, which is why I think they ended backing out. But even if it could, there are definitely better ways of handling it. The deal we ended up getting is one that benefits all sides and I wish more companies would adopt.
>In a call I was explicitly told "every company does it like this, if that's not ok you might not be a right fit for this company".
In situations like this it's helpful to have a no-management backchannel team chat group set up so you can use it synchronize a series of "nope, not doing that".
I used to work for an MSP. They billed 2-3x the normal rate for on-call to clients. We, however, were simply paid our hourly rate plus overtime. It created a perverse incentive to have as many on-call events as possible as it was very profitable for the company. They billed minimum time to clients, but we were told we could only bill for the exact minutes spent working.
I joined Prospect because my company tried to implement an unspoken on-call arrangement, whereby they would try to call me on my mobile 24/7 expecting an immediate response. I asked what the additional renumeration is for that, and they said there isn't any.
Now I'm a Prospect member, and my mobile is always on mute.
After years of iteration, here’s what our team does.
The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe.
This gets us as close to round the world coverage as we can have.
There are two people on call for each shift, each shift lasts a week.
It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal cost and spreads it out so what might be night for one person, is morning for the other and vice versa.
All of our alerts are prioritized/categorized to help prevent alert overload.
For example, an alert for a test/QA environment will not fire outside of business hours, and it has a much longer time before it’s required to be ack’ed or resolved.
There are two on-call rotas: critical and non-critical.
Critical, production-impacting, and/or client-facing alerts are dispatched to the critical rotation.
The non-critical rotation only escalates alerts during business hours, again, with a more lax timeline for acknowledgment or resolution.
People are not part of both rotas at the same time.
If there’s a big enough incident, the folks on call get to take off that next working day or the next one.
I (the manager) am on call 24/7 for escalation.
Anything that is an annoyance during on-call is a candidate for review and change.
That can be anything from thresholds to code to upgrading some IaaS/SaaS subscription. Or even straight up disabling the alert if it provides no value.
People can swap on-call days as they want.
Typically, this happens if there’s a birthday, personal event, or PTO, and it’s worked out among team members. If no one else is available, then I’ll take their shift and act as primary.
How does it work for you to be on-call 24/7 for escalation? I get that that ends up happening for many committed founders/operators/managers, but I struggle how that can be a real strategy.
Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.
I was effectively oncall 24/7 at my job at times in 2020.
I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much.
I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time.
For my part, despite my reputation, I tried to quit a few times. Not the job, the company entirely. I have never cried at work, but came close once or twice after being up for days and unwinding from a big escalation.
I think it’s a little disrespectful to resurrect someone’s comment they tried to delete. It’s their comment, and we don’t know their reasons for deleting it.
I’m paged if the two people on call both fail to ack, which is extraordinarily rare.
But what it’s there for is if the team is experiencing something that is new/novel (where I can provide some targeted guidance) or the situation is spiraling and will get worse before it gets better (where I can provide air cover).
Even in all the situations you’ve listed, I still have my phone with me.
If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors.
The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work.
I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.
I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data.
Any on-call/on-shift rotation system must be viewed through the lens of actual demand and need.
At first client, we had 3-4 developers total who shared pagers on weekly basis, as per the OP, with no undue stress or impact on their day job.
At second client, we now have multi-tier support starting with on-shift (junior but specialized ops team members who stare at computer overnight and provide immediate response), Tier 1 and Tier 2 on-call support, and multi-level escalation rotation.
And yes, there are still people who get woken up all the time always, because buck eventually stops there :-/ . Being on call sucks, as per the title. I've been in 24x7 escalation roles; I don't drink to begin with so that's not an issue, but it absolutely had significant negative impact on my social & family life, sleep and stress levels. I've spent significant effort to a) Make the system better, both in terms of more reliable application, and deeper and more self-sufficient support team tree, and b) Move myself out of the role, though that relies on success in a).
I do find it fascinating to occasionally meet very senior people, with family and social life, who are positively EAGER to be on-call and engaged for every little thing all the time always - and then, unfortunately, have same expectations of literally everybody else ("Let's all come on bridge, always, for everything, anytime")
Another data point from a startup, we have a few people also globally distributed on on-call, we use each other as backups in case as the commenter suggested you're drunk at a bar or camping. Our site doesn't break much (haven't had an incident for over a month now, maybe something happens once every 1-3 months and it's usually not severe) and we have redundancy in the schedule so we've come to not feel so neurotic about it.
> Anything that is an annoyance during on-call is a candidate for review and change.
What does that mean in practice?
Hopefully: "This woke me up last night. It's now the top priority until it's fixed so it never wakes anyone up again. Sorry product manager, your new feature will have to wait."
"This woke me up last night, but it could have been something that didn't need escalation outside of business hours."
"This woke me up last night and had I snoozed for 5 more minutes it could have been catastrophic, let's get some more proactive monitoring in place."
"This woke me up last night and it was triggered by bad user input, we shouldn't get alerted on this but more importantly, we shouldn't allow users to submit this crap."
Very rarely do I encounter alerts that are traced back to some deep architectural flaw that requires me to tango with a product manager and their roadmap.
Often times, our team escalates to the engineering lead in question and a small bug fix is slipped into the next release.
So I've been oncall at two major companies (Google and Facebook) and, at least in my experience, this covered both ends of the spectrum. Basically, Google gets it mostly right and Facebook gets it mostly wrong.
At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid when you're oncall. The amount varies depending on how important the service is and the expected response time but can easily be 5 figures a year. Oncall period varies but a week at a time varies with hopefully 8-12 people in rotation.
Too few people and people get burnt out. Even if nothing happens on an oncall shift, it's an annoyance and a restriction on what you can do. Too many and people tend to forget what to do. So with a sufficiently large team you may end up with some people in the rotation and some people not. That's why the compensation is importatnt.
Particularly large, important and mature services may enjoy SRE support. You can't throw a service over the fence and have SRE deal with it. It doesn't work that way. It typically needs to have been running for at least 6 months and SRE needs to be satisified it's sufficiently reliable, stable and monitored with a good runbook. SRE support is globally distributed and typically means 8 hour shifts during normal hours.
The owning team will often still be secondary support.
Also code has to be owned by somebody. This may be a team but when I was there (some years ago now so it may have changed) this also meant 2 actual people (not just team aliases) had to be owners. This is to avoid abandonware. This very much is a support and oncall issue.
Facebook OTOH is a dumpster fire when it comes to oncall.
Not getting paid to be oncall is (IMHO) one of the biggest mistakes. The mantra is "it's part of the job" but that responsibility is not shared equally. That's the point of compensation.
My experience at Google was that issues were relatively infrequent. What I saw at FB however was that oncall could often be the only thing you did for the week. Noisy alerts, alerts caused by issues in downstream systems that you could do nothing about or would get ignored by their oncall, a bunch of issues raised that some would just ignore until they expired (or closed just prior to going out of SLA as "could not reproduce"), etc. You may also be dealing with code that nobody owns (or, rather, nobody takes responsibility for) for features that are live.
Plus the incentive structure, at least on the product side, was to ship new features. Oncall was often treated as just extra work you have to do on top of whatever else you're doing.
Obviously I didn't see how every team did it so none of this is absolute but I did see a reasonably high number of samples.
It's also worth noting that not everything at FB is like this (eg the Web Foundation people were and I believe still are outstanding). Also, in high-visibility outage situations you have highly knowledge individuals who can and do get involved and know the right people to push.
The FB equivalent of SREs is Production Engineers ("PEs"). There are less of these and more services at FB are supported by the SWEs than at Google (IME).
I got the impression that FB processes and culture were forged when the company had less than 500 employees and they never really adjusted to the greater scale. There are a lot of things that work very well. Oncall just isn't one of them. Nor is code ownership.
I had no idea you essentially get “overtime” pay for on call at google. That’s how it should be imo. I’ve avoided on call jobs for the lack of extra pay for doing more work.
Generally 1) Mitigate if possible from your service's end, while simultaneously 2) paging the dependent service's team to mitigate/resolve; and if this happens too frequently or if it was a particularly bad incident you can push the other team to 3) create a postmortem with follow up AIs if they already didn't do so.
The deeper you go into the stack, the more reliable things tend to get, the more mature those systems tend to be and the more likely they are supported by SRE who take things very seriously.
So if you're having an issue with Spanner, first it's likely not a bug in spanner. If it's an outage, somebody has probably already been paged. But if not, paging someone responsible will be answered quickly and treated seriously.
You could've unexpectedly gone over quota on something. More often than not you can alleviate that with temporary quota while you resolve your issue (by reducing your usage, getting more permanent quota or both).
Note that the on-call bonus is not entirely known. I had managers try to put my team "on-call" for a product and after I explained to them how Google actually did it, they suddenly said "oh, it's not really on-call. You just have to be ready to answer the pager at any time and respond".
I was also on what was one of the most dysfunctional on-calls at the company- keeping several distributed clusters of unique business-critical mysql instances that failed frequently with an unreliable failover method. For some reason it was a joint on-call with a neighboring team so I was responsible for systems I didn't know about or understand but were busisness critical. At times it was fun, at times it was educational, but at times, it was the worst thing in the world for me.
I've experienced this as well and it's a sign of a really bad amanger (IME).
No manager should give you any resistance to giving away free money from the company to their team. If they ever do you know where their loyalties lie: with their management chain and producing the appearance of efficacy.
A manager should be fighting to give the team any on-call pay they're due.
Related side story: annual bonuses were (and maybe still are) calculated based on salary, level (ie target percentage) and ratings. It quickly becomes known what the base rate is so you can calculate everything. After this, your manager has a pool of extra money and a bunch of sliders for their reports. From that pool of extra money, they can distribute it evenly, weight it towards particular people, etc.
They can even take money away in this process from some people to give it to other people.
But because the formulae are all straightforward, this should be obvious. I have seen:
- Managers take away money from some people to give it to their favorites;
- Give all the extra money to one person; and
- (This is the crazy one) Not give all the money away. That is to say they'd rather not give away this free money and return it to the company. I've literally seen this happen.
That's why I mention it: any sign of a manager not giving their team everything they can should be a massive red flag.
I'd rather not go on call than get more money. That said, I kept such a good eye on prod that my team often identified problems with roll-outs in unrelated products that we didn't have control over and managed to stop them before they broke anything related to revenue. This has a neutral effect on perf.
Be On-Call is working time. Therefore you should be paid for that.
In the team that I work now, it's voluntary to join the On-Call rotation. People get paid for that. To be on-call and for any incident that they have to actively work. We have as well a partially implemented "follow-the-sun" monitoring team, but that's by accident (our team has members from West coast US, Europe and Australia, but weekends are covered by On-Call)
I have an opportunity to move into an all-remote role that would require me to be on call 24x7 for one week every 3 or 3.5 months. I asked about the frequency of incidents that required the on call person to be pages and it looked like on a bad week, it was about 7 total - a good week was 0. I’m personally torn on whether or not I’ll be okay with the on-call lifestyle so I appreciated this piece for giving me some food for thought.
That level of rotation is not a deal breaker in itself.
You can basically look at it as 3~4 rotations per year, likely 1 or maybe 2 bad weeks per year.
I've never been on a big enough team to have a rotation that wide, more typical is every 4/6/8 weeks. Beyond that teams usually build out globally and your rotation remains as often but is fewer hours per day (ie - US covers til 8pm when APAC comes in). The bigger concern arguably is how noisy weekends are?
It would be for an internal environment, so weekends would be hopefully quiet. The incident log going back a month or two showed incidents mainly in the EU and US workdays, around 5 AM Eastern to 7 PM Eastern.
Which is why I won't do it. Either hire people specifically to do it, or provide sufficient incentives so that people volunteer. Being a good "first responder" is a skill and not everyone has it or wants it.
Yeah I think most of engineering just has never heard of a night crew... And it works out to the benefit of companies, unsurprisingly.
If you want people to work around the clock, hire enough people to work around the clock. Maybe this means engineer salaries get repriced, maybe it means more and more companies figure out they don't necessarily need on-call.
All that said, the status quo probably won't change, there are more than enough engineers who either don't know better or do consider it priced in to their salaries.
The last place I worked at decided to cut the ops staff entirely and put all devs on support duty. Hilarity ensued because no one wanted to do it and the suits were baffled. They established a hard cutoff date for the transition, and at the last minute had to keep some ops on for a few months. Before that cutoff expired, they laid off almost our entire division, leaving only a fraction of untrained and unwilling staff to keep the lights on.
Even more humor, security hadn't been consulted on this and some requirements when they went to implement it. Any dev, during their 2-week support and on-call period, would have their access to the dev systems removed. Anyone not in their 2-week window would have no access to prod.
This might work at a small company with a limited number of SWE’s. Every company over 1k that I’ve worked at that followed this pattern was a shitshow when it came to incidents. Not to throw it out the window, just suggesting that it isn’t as simple as it seems here. The accountability problems surface quickly, even in very talented software engineering orgs. And of course the only thing harder than convincing a software engineer to go on call is convincing a software engineer who was hired onto a team that doesn’t do oncall to do so..
A personal horror story: I have an on-call shift that is 24/7 for one week, only 3 other people are on the shift. Alerts are frequent, noisy, and happen almost every night so you are practically guaranteed to never sleep fully that week, and the sheer breadth of services + teams we’ve accumulated and lack of any clear specificity in alerts means that I’m almost always at least somewhat confused as to if something is actually broken and, if so, how I actually fix it, even after 3 years (two of which when I lived alone during the pandemic). This was my first job out of college too, I was fully convinced that if I fucked up even once or called the wrong person I would be fired and all of the effort I put into getting this career would be meaningless.
I didn’t even get any prep or mentorship, they just suddenly put me on-call during a major product launch. No extra pay or time off btw, just gotta continue work if you were up all night trying to fix some thing with vague priority & vague symptoms (too much latency on random offline service I’ve never heard of that turns out to be a dev experiment, latency being caused by a laggy database that you find out by finding some random message in splunk and regexing it out and into a graph).
I’m definitely a changed person after it, I don’t really… react as much anymore and flinch every time I hear a default iPhone text notification or ringtone. I don’t know how to fix it either—I don’t know if our team has enough people to spread the load out and I can’t think of any better way to keep track of failures in this labyrinth of services, and onboarding people to the point where they can actually take an additional shift is usually 8-12 months. Even experienced people still get ambushed by new services with zero documentation.
Pros though: I don’t really experience much stress or uncertainty anymore in hard situations, and I seem to be much better at problem solving! I’ve also managed to keep my prestigious job with life-changing pay, which feels much more personally fulfilling than coasting at Google (even if it’s for the wrong reasons).
I worked at a place in the 2000's where our main application leaked like a sieve due to not releasing memory from a C++ framework, and two people had to take turns every other night restarting the app (actually it was chopped up into 20 different apps that had to be started in order by hand) every two hours or so. I can't imagine they got much sleep on their night.
A major issue with on-call, and certainly one I've encountered multiple times, is the high likelihood of moral hazard - the people who are responsible for addressing incidents are not the same people who designed and maintained the system at fault. This results in the former team feeling powerless to put out fires which could have been prevented by more robust design, and the latter team having no incentive to improve reliability.
SRE gets this right, at least in theory, by requiring that all production systems be reviewed and approved, including observability and incident management procedures, prior to entering service. This ensures that there is some shared responsibility across teams for maintaining uptime.
>> A major issue with on-call, and certainly one I've encountered multiple times, is the high likelihood of moral hazard - the people who are responsible for addressing incidents are not the same people who designed and maintained the system at fault. This results in the former team feeling powerless to put out fires which could have been prevented by more robust design, and the latter team having no incentive to improve reliability.
The Amazon approach was (still is?) to have the team that develops and deploys the software to be responsible for the on-call rotation for that system.
If you are developing software for such a team, it gives you a direct reason to make sure everything is designed and tested well before it is deployed to production--you (or a teammate) will be answering the early morning alert to fix it if it is not.
A direct feedback loop like that is remarkably effective to prevent the moral hazard and ensures direct accountability when buggy software gets deployed.
The most attractive thing to me in that SRE handbook is the error budget. I don't mind poking at your thingy but I'm only going to do it a few times before it becomes your problem instead.
> SRE gets this right, at least in theory, by requiring that all production systems be reviewed and approved, including observability and incident management procedures, prior to entering service.
Both doing, and being subjected to reviews suck balls. I’d much rather be on call 24/7 to fix the problems I caused myself.
Or it's incentivized in the wrong direction, in the case of something like an MSP that bills more for out-of-hours calls so it's actually profitable to have events happen.
I'm one of the few persons I ever heard of that actually enjoyed being on-call. I believe it goes with my puzzle problem solving mentality to an extent. Being randomly challenged with a problem to look at where you might not know the solution, simply excites me.
Combining on-call duty with an approach of weeding out repeating issues, build better systems and ensuring that unnecessary calls don't happen is key of course, being woken up 25 times for silly predictable errors is pointless and draining.
And finally having an employer that doesn't expect you to be in at 8am if you've been up all night is also very important, catching up on sleep is necessary to manage your balance and health. But given this freedom, I totally dig it. :)
> And finally having an employer that doesn't expect you to be in at 8am if you've been up all night is also very important, catching up on sleep is necessary to manage your balance and health. But given this freedom, I totally dig it. :)
Check the labour laws in your country, I'm pretty sure expecting people to be in at 8am after working on an incident during the night as part of your on-call is illegal.
In any normal country there are laws to ensure employees get enough rest every day.
On-call is even worse for people with disabilities. I quite literally can't do it unless I stop taking my antipsychotic.
Under ADA, I can not be placed on call, regardless of policy, nor can I be discriminated against for that. On-call is not an essential function of being a software developer, with very few exceptions—all of which have nothing to do with "policy" or "fairness".
Needless to say, companies (and some coworkers) really don't like this.
Not to detract from you comment, but you don't have to have a disability. Some people simply cannot do on-call, disability or not.
I had a co-worker at a previous job, he did two or three on-call rotation and told our boss that he couldn't do it. Mentally it's simply to much for him, especially outside business hours where he felt alone with to much responsibility. In terms of abilities and qualifications he was absolutely able to do the job. Nobody complained or got angry with him over it, because everyone could relate.
At the other end of the scale I had another co-worker, in a more complicated scenario who absolutely didn't care. The payment for the on-call shifts was very good, so he just grabbed as many as possible. He would just take his laptop golfing, no problem. His reasoning: Either he'd know how to fix the problem, if not he'd just to call someone else and hand of the incident.
I didn’t even think about that, but absolutely. I’m not a single parent, but my wife works in retail and just those late hours make daycare pick ups, late afternoon meeting, on-call and incidents a major hassle.
Unless you've tested this theory in court it might not be true. It's almost certainly not as cut and dry as you make it seem. Many companies put the same people oncall who write the code, meaning it literally is an essential function of a software developer to provide oncall support. You'd have to argue in court that it's not really essential but it would be situationally dependent.
That said, I'd hope most places would be willing to accommodate you. Places I've worked have always treated oncall as a kinda optional "right thing to do". I've never seen anyone punished for missing an alert. You'd have a good argument if that were the case at your company but that approach to oncall is not universal.
That would be one of the exceptions. For example, high-frequency trading firms always need developers on call while they are actively operating. Keeping those systems running correctly is essential to their role in the company. Same with small companies who have no other staff, assuming they even have enough employees (15) to be under ADA. :)
For the more common scenario of on-call rotation, it would be very difficult to make that argument because other people can take up the disabled person's shifts.
We are even in the process of spinning up some product-only oncall for issues that don't need dev "escalation" (e.g. most commonly some customer messed up their credentials or their API token expired end-of-day, resulting in some higher error rate).
After being a SWE for a decade in Europe, I have never heard of anyone in my network who needed to be on-call. Is this a US thing or only for devops?
Why would a software engineer need to be on call ever? That just means the CICD/testing/validation pipeline sucks.
Well in finance/trading, you need 24/5 or even 24/7 human monitoring. If you don't respond to that alert, you can wake up and see that the system just has lost tens of millions. Core engineers usually take on call duties, as fast diagnostics and response is critical.
What I have seen in finance/trading is that the required 24/7 monitoring is done by staffing 24/7 in pre-arranged shifts, not having this done by people being on-call while sleeping between their regular 9-5 workdays.
I'm sure there's companies in Europe that have on-call shifts, but I've had several jobs now and the only company that ever asked my team to do on-call outside of regular working hours was a US company and we had to explain to them how German labour laws work (specifically that you have to pay people for being on-call and that there are legally required rest periods).
My last gig, on call worked well, I thought, for a few reasons: it was our services that we wrote, it was 1 wk out of 6 that you were on call, we heavily prioritized fixing unactionable alerts and automating fixes -- every alert had a runbook entry that described the non-automated fixes, and while on call your or sprint commitments were not counted,
That last point was very nice as it meant you could work on whatever you felt was most important for quality of life improvements all week long while not fielding on call issues. This meant that I looked forward to on call.
It is OK being on call on your servers and your software.
I think it is also why less and less people having stuff on premise and going for SaaS/Cloud solutions.
If someone wants to have my software on their servers and me not having any access better they have dedicated person for running it and I don't care even if they pay $1000 per hour - I am still not dealing with server I don't know talking on the phone with admin that has no clue how my software should be configured.
Honestly, that last point seems like something that would make on-call extremely palatable. Essentially acknowledging "on-call sucks; in return, here's the latitude to work on whatever you happen to think is important/interesting"
I enjoy being on call. I have heavy ADHD, so I am not a fan of routine, well-defined time activities and deadlines, but if I get this call at 2AM, I feel intense adrenaline rush, and I feel like a hero of the night saving the prod.
I'd like to learn more about that, it seems relevant to my situation. Some context (I'm not implying that this applies to you): we have an (well above average) capable engineer with a hero complex. In the most difficult of situations, he'll dive in and save the day (or night) where everyone else fails. He's also the one in the team who resisted standardisation and automation the most which, in a way, lead in the past to situations where his unique skills were required.
Yes, this looks like my case. This is why I had to quit the corporate workforce and start working as a consultant — it provides me enough novelty and suits well to my strengths. I realize that in a regular corporate environment, I can be a liability instead.
Thank you. Do you have advice for me how to create a more welcoming and productive work environment for my colleague?
Edit: especially with respect to the automation/standardisation approach we've taken?
I wish all PMs would go on call for a week at least. The OP's stint with on-call will be quite useful in his career since he can better intuitively view infrastructure more holistically. Then if the group is doing sprints, the "firefighting" will more easily get prioritized.
281 comments
[ 3.2 ms ] story [ 281 ms ] threadFor companies that paid you overtime, it often starts with guilt, "well the rest of the team did it last week/month/year, it's your turn". If they don't incentivize you enough to do it, don't do it.
On top of this, we have a rotation so each of us is only on-call one week out of maybe every four or five. And although I agreed to it mostly because it was a condition of the job and I wanted the job more for learning how the team worked than I cared about the money, the compensation for being on-call is actually pretty good even if nothing happens. And if on-call lands on a holiday, we get the holiday time as vacation days to spend later.
So overall, while I would prefer not to be on-call, I feel like our team implements it about as well as can reasonably be expected. I expected it to drive me crazy, but it actually hasn't yet.
The other factor was that a "call-out" was not completed until the root cause was fixed.
I believe the real reason for the compassionate arrangements was that the owners of the business were former engineers and were even available to escalate calls to them if you got stuck. Our personal phones had everybody else's personal numbers in the address book, but we were never permitted to give them out to clients. Clients only had access to the "hot potato" phone numbers, which also received the various paged alerts, etc.
1. There is a rotation.
2. There are few pages, preferably the median should be 0 per week.
3. Spurious / non-actionable alerts get fixed right away (with very high priority)
4. You're not up more than 1 week per 1-1.5 month.
5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime.
As with most things, the core idea is not bad, it's the execution that matters.
This is important. If you are on-call, you do not have the freedom that you would otherwise have and you should be compensated for this.
Usually those hours are harder to predict compared to working hours so it is much easier to pay for them outside of typical monthly salary.
And sure, it is priced in because the companies can get away with it. This doesn't mean it is right or fair as the price is always going to be in the company's favor and rarely fair to the employee. For example: My brother worked for a US railroad. He didn't have a set schedule. Instead, he got 10 hours rest after a shift and then he was on call. They only closed down on Christmas and New Year's. You were expected to do this on-call work perpetually. The money was good for the area as were the benefits. They advertised in depressed areas without much opportunity, so it made it easier to prey on folks that will accept the poor treatment. I'm pretty sure having to pay folks for each of those hours would change the behavior of the company in the employee's favor.
So yeah, even one hour of 'on-call' should be paid extra, outside of your basepay. Even for companies and industries that aren't being actively evil so that it doesn't happen in the future.
My contract says that if on-calls are needed, I might have to be in the rotation. This clause increases my pay rate even if I'm not on-call.
If however I am actually on-call, I am paid more. And if the on-call rings, I'm again paid more on top of the on-call period. And as the French law mandates 11 consecutive hours of rest, if the on-call rings in the middle of the night, I'll usually come to work later the day after.
If you don't have advantages for being on-call, you're the one being taken advantage of.
* For every hour you're available for on-call you get your regular hourly rate.
* For every incident, which involves you working, you receive twice your hourly salary for the duration.
So companies based her tend to have that as standard, though I'm sure some companies would pay more to stand out.
Being on-call while you do not get called upon is 24 / 7 work because you have to live your entire life around being available.
Like the blog post mentions, grocery shopping has implications because if you happen to have an on-call event while shopping it means leaving your cart to run to your car to address the situation because you can't leave your house without your work laptop.
It means never going to the beach alone on a Saturday because while you could bring your laptop with you, if you go swimming and there's an on-call event you can't address it because you have no way to get notified while you're swimming in the ocean.
It also means going to the movies with an expectation that if you get called 15 minutes into the movie you're leaving. Likewise, if you're mid-date and get called you're out of luck.
It means never enjoying being able to walk around while being disconnected from the world. It means if you're at your mom's funeral giving a eulogy you leave mid-speech to address PagerDuty.
Then there's knowing at any given second your phone can notify you of an event and you have to put the volume at maximum and place it right next to your face every night with an expectation that you could be woken up at any second.
You should not be on-call while you're at your mom's funeral, period.
And just to pile on that even more: also dreading that you might not wake up if the page/call comes during the heaviest hours of your sleep.
It's happened to me, I felt awfully guilty for not waking up after a gruelling work day and some pages early in the evening. I was tired and had been asleep for around 2 hours, didn't wake up and the escalation policy took it up to my manager... I didn't get reprimanded or had any bad consequence from it, still the guilt made me feel like a failure and increased my anxiety when I'm on-call.
I don't know if this will help with your mindset at all, but as someone who's been listed as the secondary if the first responder doesn't answer: I really wouldn't beat yourself up over it. Self-correction is fine, but try not to let it drag you down too far. We have the same worries too. Our worries are usually "Crud. If I don't pick up, then it's either going to (big boss) or the customer is going to be firing off nasty emails in the morning and dragging me into some stupid meeting." The cause of why we're being paged doesn't really come into play. At most, I might just send a text or whatever to the first responder, to make sure that they're okay. "What if they aren't responding because they were in a car accident?"
If you have a good manager - and it sounds like you do, since you weren't reprimanded for performing a fundemental human function - then they are probably doing their best to look out for you and your interests. When people talk about "not being the boss" or "working as a team" or whatever, generally what they're trying to get at is that "They have your back". They've been in your position and have probably felt that same crush of emotion and worry that you might deal with. They are meant to be the final filter, in a hierarchy of filters, to protect you from those outside elements which ruin your ability to work effectively.
If you put in a sincere effort at your job, you have nothing to feel guilty over. That might be easier said than done, but it really is true. It just might take some time and practice to be able to forgive yourself when you make a mistake.
And if you feel like you could be doing a better job at this, or at that, then you just need to permit yourself the time to improve on that thing, and to remind yourself that you are allowed to mess up from time to time. It's about how you learn from those things.
The only person I ever had to formally reprimand for on-call policy wasn't the one who failed to answer on time, it was the person who, three times, acked and didn't say or do anything else only because they didn't know how to fix it. In all cases I ended up getting pinged by someone else (head of our support team, or a colleague working an odd schedule), after the situation significantly worsened.
Higher tiers are there for a reason; don't feel bad if you've used them, only if you're expecting to use them.
(On the other hand, feeling guilty for making a mistake is also human, and to some degree something that makes me want such a person on my on-call team. People who feel better when they do a better job will do a better job!)
On-call should be at most one week in four-six. Moreover, with a healthy on-call culture (where stuff is fixed, and alerts happen rarely in practice), usually you can pass/swap on-call to others for an evening, or for an afternoon, or for a weekend, as almost always there is somebody who's plan is "sitting at home" and nobody minds having the pager in such circumstances if it almost never pages outside of working hours.
I wonder if this is an American attitude that exists primarily because we (collectively) have allowed our employers to demand this of us?
Personally, I hate being on call. Most weekends, I spend at least 6 hours cycling. Sometimes significantly more. I go camping regularly. I go kayaking regularly. If I don't spend time outdoors, my mental health declines (really, ask my wife, she kicks me out if I sit around the house too long because I turn into a cranky butthead). Most evenings after work, I run/cycle/hike. I have to run errands. Walk the dog (usually 2 miles). This is all part of my normal "not doing anything special" time at home. None of which is easily doable if I'm on call.
Fortunately, I've managed to build a career in a place where on-call rarely exists.
And holy moly! One whole week of being on-call is way too much.
There are (or should be) some minimum standards of rest time that every human being must get, and being on-call is not it, every employee must get an opportunity to fully disconnect for non-trivial time during every single week. You do your shift, and then get at least 12 hours off (with your phone off) before being available for work or work calls again.
And if you're getting paged outside of work hours, then there's zero obligation to answer your phone.
There's no middle ground. If the employer says that these aren't working hours, they have zero right to ask what you're doing at these hours, much less put any conditions on it; it's your right to spend that time fishing in a remote lake with no cell phone service or go on a date or whatever and not even explain anything without any reprimand when you arrive at your scheduled start of work time; and if they want you to commit to a shift, well, "a shift" where employer tells you what to do (e.g. do not go fishing to a remote lake and don't sleep for 12 hours with your phone off) is by definition work hours.
It is quite plausible that most of the times most employees will pick up the phone and solve reasonable issues, but that's an extra courtesy from the employee, going beyond what you can demand or expect; but the moment you start to require that, or ask an employee to precommit that they will definitely be monitoring their phone for rapid response, that means you're effectively assigning those hours as work hours.
It's understandable that being on-call can be very light work in many cases (not all - quite a few counterexamples in this discussion), so you can agree on different compensation for them, but those definitely are work hours (they definitely aren't non-work hours, and there is no middle ground) and thus any rules on length of shifts and rest between shifts can and should apply also for on-call hours.
I don't see how you can declare what an employer/employee are allowed to agree to. Working an on-call is completely reasonable for a salaried employee. If the terms changed after you were hired and you're bound by a contract (and cannot leave without penalty for a certain timeframe) then things would be very different. But the majority of the HN audience is in the U.S. where employment is generally at will.
Okay, I am coming from a non-US perspective where it's obvious that you can declare what employer/employee are allowed to agree to - "employee rights" means those things which are unalienable and nonnegotiable.
I'll simply quote the universal declaration of human rights "24. Everyone has the right to rest and leisure, including reasonable limitation of working hours and periodic holidays with pay." - not only do countries have the right to intervene in employer/employee contracts, they have a duty to do so. If local employment law permits companies to require salaried employees to work 168 hours a week, then that law is literally enabling violation of human rights and should be changed.
In any case I suppose we have different expectations on what's reasonable. If you think a handful of pages per year from my consentual employer is infringing on my fundamental human rights then there's not much I can say to convince you otherwise.
It's not about the number of pages you get, but about the differing expectations on boundaries between your employer and your private time. I do have a strong expectation that employers should not be able to set any restrictions of what people will do in their non-work time, or even ask whether the employer will be out-of-service this weekend, in my opinion that is crossing a line.
As I said, it's entirely reasonable if you get a handful of pages per year and service them - I also have been in roles where I got a handful of night calls per year and happily did what was useful, and I'd expect that in most cases that's okay for most people. However, if it becomes a requirement where I or you must be "on alert" for a whole week, then IMHO that's not reasonable anymore; I answered those calls but I had no duty to be ready for them and abstain from activities that make me unreachable, and if I had a contractual obligation to do so that would be unreasonable (and also a void, unenforceable clause violating employment law). And if you did not volunteer for it but an employer "pushed" this unreasonable requirement on you (e.g. as a condition for employment) then I would actually say that this violated your human rights even if you got zero pages per year; employees should have the practical right to freely decide how they spend their non-work time during the week (and they should have appropriate non-work time) no matter what contracts they sign.
24 hour shifts? Sounds like a real worker's paradise. I'll keep my 7-8 hour days with bi-monthly on-call over that, thanks. I'm glad you aren't legislating where I live; I am much happier with the choices and trade-offs in my life than what you or any euro-crat could dream up to "protect" me.
> if you did not volunteer for it but an employer "pushed" this unreasonable requirement on you (e.g. as a condition for employment)
You have a very different concept of what voluntary and self-agency mean than I do.
I think we could all stand to learn a bit from the Finnish and the Norwegians.
in my experience, most employer seem to think responding within 30minutes or so is doable, and i tend to agree with them. Usually makes it possible for you to just do things around the house/ in your life, and not having to rush back home when you run into an oncall issue.
I can't imagine less than 30 minutes.
They were most displeased when I declined this opportunity.
Additionally, I'll never accept on-call with 15 minutes from alert to being on a computer again. It's just too limiting and disruptive of my life. Have to bring the computer everywhere. Any dinner or social event can be instantly ruined. A workout becomes meaningless. I remember doing a swim and having to check my phone every 5 minutes. It's mentally exhausting and frankly not worth the pay.
This is impoprtant because if you are on-call, you are working, and if it is during a holiday, then that is one less holiday for you.
This could actually be pretty good too. People would be much more willing to cover for others on holidays
I have a fair bit of experience in teams with developers hating oncall and the critical issues happen in two camps generally:
- In some cases the org was absolutely open to give them time to resolve issues and automate stuff away (PMs were proposing months for fixes only, and cleaning up boards, etc), there just was no interest until enough escalation happened (creating a massive conflict between ops and devs in the meantime). Even offering to do the work was met with a "stay in your lane" kind of response and no collaboration at all.
- In others the company just did not care, deprioritized tickets until things blew up completely, then finger pointing started and all that nice toxic bullshit.
That's insane. That's around 20% of your life.
That seems excessive if you're expected to be able to log into your work system within X minutes.
Having to be essentially home, near a computer, 25% of the time (1 week out of 4) is a pretty heavy burden, especially for people who prefer to be out, rather than home.
In interviews, always ask them:
1. How often is someone on call (typical was one week every 6-8 weeks).
2. While on call, how often do you get paged after hours?
3. What do you do to reduce the number of pages?
I suspect I've been rejected for merely asking the second and third question, but that's good!
The worse part of oncall is the control of your life it has. for one week I can't do anything I would normally do. (if your company actually compensates for this, let me know where i can a apply, or better, if it doesn't have oncall at all!) Of course managers are never oncall 24/7. The worse is they give the excuse well im on call all the time by default since im the one manager. But theyre not reorganizing their life and putting their off work hobbies on hold becasue of it are they?
> a monitoring change that fixes some flaky alert that might page somebody about once every six weeks.
These kind of things suck. I was on a team where we had tons of these, 10 alerts like this mean your getting pages all the time. No single alert is worth the time investment. Worse was a manager insisted there will always be a base line of alerts that go off and we will just live with it.
Teams never seem to understand how to alert on stuff. Ive been paged for things going off, that might indicate a problem, then you get stuck sticking around because someone else wants to just wait and see what happens. "We should just be cautious" Its impossible to push back on these things, your just going against someones gut feeling, like maybe one day we will want to know, and everyone needs to protect them selves.
> This can obviously be fixed with good documentation, but in reality, no one has good enough docs.
One problem I've witnessed related to documentation about on-call issues is the over reliance on the SOP concept. They only commit to one level or one pass of analyzing the issue. They do not future drill down, either by linking to other notes or reviewing the issues deliberately. It's like they read about the 5 Whys and decided why not just 1 why.
From an ops person: if an alert does not have: - clear, provable impact on customers (internal/external) - clear documentation (e.g. runbooks) on how to solve it
It should not be an alert. I took this path (successfully) when trying to remove spurious alerts that existed only for the ego of someone (most absurd example, something that started complaining when p99 for some endpoints went >500ms and happened every day when we downscaled the ASGs because business hours were over. No clear path to resolution, and impact was a couple pages opened a bit slower sure - but the number of customers using those pages after hours was <1%!
It sucks, definitely, but the best way to go around those alerts is to prove they're pointless or a waste of time or can be automated around and should automated around (and I've seen so many servlets leaking memory triggering OS alerts for OS teams or spawning infinite threads and never cleaning up after themselves...).
If the company does not want to do it, and pushes back, I would recommend starting to look for another company. It's sad, but it is what it is. 99.99% of software does not need a follow the sun rotation (or people damned to night shifts), just a bit of thought about what happens when things fail.
Sometimes that works fine, and you'll get no alerts all week, but incidents tend to cluster. If something has changed that caused an incident odds are it's going to have knock on effects, and you'll see more alarms over the course of a week. With a weekly rota you end up with one person handling that, who by the end of the week is completely destroyed.
Anywhere I've been responsible for setting up an on-call rota I've instead gone for daily rotations. That means if you were up in the night last night, someone else is going to be in the night tonight. It also means if nothing happens you don't have to spend an entire week either cancelling plans or lugging a laptop around with you just in case.
The initial wording was very restrictive, like 5 minute acknowledgement time and 15 minutes at-laptop. 24/7 for 7 days. They tried to have this implemented without any extra remuneration or perks for the on-call engineer.
On top of it possibly being very illegal, it seems very immoral to spring something like that on people that did on agree to it when they took the job.
I fought for it and I got them to change their policy in 2 mostly meaningful ways:
- It's an opt-in method
- On-call engineers get paid extra for just being on-call and get extra time off whenever they need to actually do something.
This makes sure that you only get people actually willing to do it and there is an incentive. I think it's been quite a successful program!
Luckily I didn't need to get them involved, but in the UK there are unions starting to form for tech workers, I suggest you join one like https://prospect.org.uk/tech-workers
I don't understand what "force" means in this context - the conversation went something like "I have commitments outside of work" and that was that. I mean, there was a back and forth, but yeah, at the end of the day I took the job knowing I'd be available for the hours they wanted when I took the job.
In a call I was explicitly told "every company does it like this, if that's not ok you might not be a right fit for this company".
I don't think anyone is the right fit for any relationship in which one side attempts to change the parameters without consultation.
It just doesn't make any logistical sense. You may as well ask a teenager doing a weekend job to come in on Tuesday. They're at school.
UK employment law doesn't really permit these sorts of shenanigans.
In situations like this it's helpful to have a no-management backchannel team chat group set up so you can use it synchronize a series of "nope, not doing that".
it's like as long as its not me, i dont care how much your suffer.
Now I'm a Prospect member, and my mobile is always on mute.
The team is remote and distributed across multiple time zones ranging from West Coast US to Western Europe.
This gets us as close to round the world coverage as we can have.
There are two people on call for each shift, each shift lasts a week.
It will typically (but not always) be one person from US and one person from UK/EU. This helps reduce the single personal cost and spreads it out so what might be night for one person, is morning for the other and vice versa.
All of our alerts are prioritized/categorized to help prevent alert overload.
For example, an alert for a test/QA environment will not fire outside of business hours, and it has a much longer time before it’s required to be ack’ed or resolved.
There are two on-call rotas: critical and non-critical.
Critical, production-impacting, and/or client-facing alerts are dispatched to the critical rotation.
The non-critical rotation only escalates alerts during business hours, again, with a more lax timeline for acknowledgment or resolution.
People are not part of both rotas at the same time.
If there’s a big enough incident, the folks on call get to take off that next working day or the next one.
I (the manager) am on call 24/7 for escalation.
Anything that is an annoyance during on-call is a candidate for review and change.
That can be anything from thresholds to code to upgrading some IaaS/SaaS subscription. Or even straight up disabling the alert if it provides no value.
People can swap on-call days as they want.
Typically, this happens if there’s a birthday, personal event, or PTO, and it’s worked out among team members. If no one else is available, then I’ll take their shift and act as primary.
Are you never off-grid for a bit, or drunk in a bar, or just on a real no-work vacation? There seem to be situations where being on call just isn’t feasible.
I barely noticed the pandemic. I never strayed far from my computer. Also, yes, I tried not to drink much.
I certainly learned what my limits are. People think I am a pretty good engineer (not amazing) but what I am known for is being able to keep that level of performance up for a long time.
For my part, despite my reputation, I tried to quit a few times. Not the job, the company entirely. I have never cried at work, but came close once or twice after being up for days and unwinding from a big escalation.
But what it’s there for is if the team is experiencing something that is new/novel (where I can provide some targeted guidance) or the situation is spiraling and will get worse before it gets better (where I can provide air cover).
If I’m going to be out of cell coverage (e.g. a plane ride, or in the countryside with spotty Internet) or simply want to be left alone, I usually plan for that in advance and do a combination of: 1) scaling back our risk exposure by rescheduling work (which requires you to have a good understanding of the business, its needs, and its timelines) and 2) shoring up the bits I feel most weary about through code, documentation, tooling, and/or contractors.
The same goes for the team SMEs: reschedule where I can, crosstrain where I can’t, get headcount where none of the prior work.
I’ve been on call since 1999. I had to figure out a pattern that worked for me (and my family) but wouldn’t result in a life that was boring or worse, one that I resented.
Depends heavily on how stable app/system is.
I'm from PeopleSoft / ERP world. The same PeopleSoft HCM base application would generate one off-hours pager alert a month at one client; and half a dozen a night at another client; due to different customization/implementation/complexity of business logic and data.
Any on-call/on-shift rotation system must be viewed through the lens of actual demand and need.
At first client, we had 3-4 developers total who shared pagers on weekly basis, as per the OP, with no undue stress or impact on their day job.
At second client, we now have multi-tier support starting with on-shift (junior but specialized ops team members who stare at computer overnight and provide immediate response), Tier 1 and Tier 2 on-call support, and multi-level escalation rotation.
And yes, there are still people who get woken up all the time always, because buck eventually stops there :-/ . Being on call sucks, as per the title. I've been in 24x7 escalation roles; I don't drink to begin with so that's not an issue, but it absolutely had significant negative impact on my social & family life, sleep and stress levels. I've spent significant effort to a) Make the system better, both in terms of more reliable application, and deeper and more self-sufficient support team tree, and b) Move myself out of the role, though that relies on success in a).
I do find it fascinating to occasionally meet very senior people, with family and social life, who are positively EAGER to be on-call and engaged for every little thing all the time always - and then, unfortunately, have same expectations of literally everybody else ("Let's all come on bridge, always, for everything, anytime")
Sometimes the answer is yes. Sometimes the answer is no. If it’s down already it can’t get worse.
To be fair, I was the first (and only) point of escalation.
Oh yes it can.
What does that mean in practice?
Hopefully: "This woke me up last night. It's now the top priority until it's fixed so it never wakes anyone up again. Sorry product manager, your new feature will have to wait."
"This woke me up last night and had I snoozed for 5 more minutes it could have been catastrophic, let's get some more proactive monitoring in place."
"This woke me up last night and it was triggered by bad user input, we shouldn't get alerted on this but more importantly, we shouldn't allow users to submit this crap."
Very rarely do I encounter alerts that are traced back to some deep architectural flaw that requires me to tango with a product manager and their roadmap.
Often times, our team escalates to the engineering lead in question and a small bug fix is slipped into the next release.
At Google, a new service has to be supported by the team that developed it. There'a an extensive launch checklist that includes monitoring, having a runbook, etc. Here's the most important part: you're paid when you're oncall. The amount varies depending on how important the service is and the expected response time but can easily be 5 figures a year. Oncall period varies but a week at a time varies with hopefully 8-12 people in rotation.
Too few people and people get burnt out. Even if nothing happens on an oncall shift, it's an annoyance and a restriction on what you can do. Too many and people tend to forget what to do. So with a sufficiently large team you may end up with some people in the rotation and some people not. That's why the compensation is importatnt.
Particularly large, important and mature services may enjoy SRE support. You can't throw a service over the fence and have SRE deal with it. It doesn't work that way. It typically needs to have been running for at least 6 months and SRE needs to be satisified it's sufficiently reliable, stable and monitored with a good runbook. SRE support is globally distributed and typically means 8 hour shifts during normal hours.
The owning team will often still be secondary support.
Also code has to be owned by somebody. This may be a team but when I was there (some years ago now so it may have changed) this also meant 2 actual people (not just team aliases) had to be owners. This is to avoid abandonware. This very much is a support and oncall issue.
Facebook OTOH is a dumpster fire when it comes to oncall.
Not getting paid to be oncall is (IMHO) one of the biggest mistakes. The mantra is "it's part of the job" but that responsibility is not shared equally. That's the point of compensation.
My experience at Google was that issues were relatively infrequent. What I saw at FB however was that oncall could often be the only thing you did for the week. Noisy alerts, alerts caused by issues in downstream systems that you could do nothing about or would get ignored by their oncall, a bunch of issues raised that some would just ignore until they expired (or closed just prior to going out of SLA as "could not reproduce"), etc. You may also be dealing with code that nobody owns (or, rather, nobody takes responsibility for) for features that are live.
Plus the incentive structure, at least on the product side, was to ship new features. Oncall was often treated as just extra work you have to do on top of whatever else you're doing.
Obviously I didn't see how every team did it so none of this is absolute but I did see a reasonably high number of samples.
It's also worth noting that not everything at FB is like this (eg the Web Foundation people were and I believe still are outstanding). Also, in high-visibility outage situations you have highly knowledge individuals who can and do get involved and know the right people to push.
The FB equivalent of SREs is Production Engineers ("PEs"). There are less of these and more services at FB are supported by the SWEs than at Google (IME).
I got the impression that FB processes and culture were forged when the company had less than 500 employees and they never really adjusted to the greater scale. There are a lot of things that work very well. Oncall just isn't one of them. Nor is code ownership.
How does Google deal with issues caused by downstream systems causing alerts?
So if you're having an issue with Spanner, first it's likely not a bug in spanner. If it's an outage, somebody has probably already been paged. But if not, paging someone responsible will be answered quickly and treated seriously.
You could've unexpectedly gone over quota on something. More often than not you can alleviate that with temporary quota while you resolve your issue (by reducing your usage, getting more permanent quota or both).
A big part of this is that it's a cultural thing.
I was also on what was one of the most dysfunctional on-calls at the company- keeping several distributed clusters of unique business-critical mysql instances that failed frequently with an unreliable failover method. For some reason it was a joint on-call with a neighboring team so I was responsible for systems I didn't know about or understand but were busisness critical. At times it was fun, at times it was educational, but at times, it was the worst thing in the world for me.
No manager should give you any resistance to giving away free money from the company to their team. If they ever do you know where their loyalties lie: with their management chain and producing the appearance of efficacy.
A manager should be fighting to give the team any on-call pay they're due.
Related side story: annual bonuses were (and maybe still are) calculated based on salary, level (ie target percentage) and ratings. It quickly becomes known what the base rate is so you can calculate everything. After this, your manager has a pool of extra money and a bunch of sliders for their reports. From that pool of extra money, they can distribute it evenly, weight it towards particular people, etc.
They can even take money away in this process from some people to give it to other people.
But because the formulae are all straightforward, this should be obvious. I have seen:
- Managers take away money from some people to give it to their favorites;
- Give all the extra money to one person; and
- (This is the crazy one) Not give all the money away. That is to say they'd rather not give away this free money and return it to the company. I've literally seen this happen.
That's why I mention it: any sign of a manager not giving their team everything they can should be a massive red flag.
In the team that I work now, it's voluntary to join the On-Call rotation. People get paid for that. To be on-call and for any incident that they have to actively work. We have as well a partially implemented "follow-the-sun" monitoring team, but that's by accident (our team has members from West coast US, Europe and Australia, but weekends are covered by On-Call)
I've never been on a big enough team to have a rotation that wide, more typical is every 4/6/8 weeks. Beyond that teams usually build out globally and your rotation remains as often but is fewer hours per day (ie - US covers til 8pm when APAC comes in). The bigger concern arguably is how noisy weekends are?
Often it becomes a tool to force people to work a whole lot more.
Which is why I won't do it. Either hire people specifically to do it, or provide sufficient incentives so that people volunteer. Being a good "first responder" is a skill and not everyone has it or wants it.
If you want people to work around the clock, hire enough people to work around the clock. Maybe this means engineer salaries get repriced, maybe it means more and more companies figure out they don't necessarily need on-call.
All that said, the status quo probably won't change, there are more than enough engineers who either don't know better or do consider it priced in to their salaries.
Even more humor, security hadn't been consulted on this and some requirements when they went to implement it. Any dev, during their 2-week support and on-call period, would have their access to the dev systems removed. Anyone not in their 2-week window would have no access to prod.
I didn’t even get any prep or mentorship, they just suddenly put me on-call during a major product launch. No extra pay or time off btw, just gotta continue work if you were up all night trying to fix some thing with vague priority & vague symptoms (too much latency on random offline service I’ve never heard of that turns out to be a dev experiment, latency being caused by a laggy database that you find out by finding some random message in splunk and regexing it out and into a graph).
I’m definitely a changed person after it, I don’t really… react as much anymore and flinch every time I hear a default iPhone text notification or ringtone. I don’t know how to fix it either—I don’t know if our team has enough people to spread the load out and I can’t think of any better way to keep track of failures in this labyrinth of services, and onboarding people to the point where they can actually take an additional shift is usually 8-12 months. Even experienced people still get ambushed by new services with zero documentation.
Pros though: I don’t really experience much stress or uncertainty anymore in hard situations, and I seem to be much better at problem solving! I’ve also managed to keep my prestigious job with life-changing pay, which feels much more personally fulfilling than coasting at Google (even if it’s for the wrong reasons).
SRE gets this right, at least in theory, by requiring that all production systems be reviewed and approved, including observability and incident management procedures, prior to entering service. This ensures that there is some shared responsibility across teams for maintaining uptime.
https://sre.google/sre-book/being-on-call/
The Amazon approach was (still is?) to have the team that develops and deploys the software to be responsible for the on-call rotation for that system.
If you are developing software for such a team, it gives you a direct reason to make sure everything is designed and tested well before it is deployed to production--you (or a teammate) will be answering the early morning alert to fix it if it is not.
A direct feedback loop like that is remarkably effective to prevent the moral hazard and ensures direct accountability when buggy software gets deployed.
Both doing, and being subjected to reviews suck balls. I’d much rather be on call 24/7 to fix the problems I caused myself.
The second one weird trick is to legitimately ask people about the on call experience. Again. And again. And again.
The reason nothing change is because noone is incentivised to change it.
Combining on-call duty with an approach of weeding out repeating issues, build better systems and ensuring that unnecessary calls don't happen is key of course, being woken up 25 times for silly predictable errors is pointless and draining.
And finally having an employer that doesn't expect you to be in at 8am if you've been up all night is also very important, catching up on sleep is necessary to manage your balance and health. But given this freedom, I totally dig it. :)
Check the labour laws in your country, I'm pretty sure expecting people to be in at 8am after working on an incident during the night as part of your on-call is illegal.
In any normal country there are laws to ensure employees get enough rest every day.
Under ADA, I can not be placed on call, regardless of policy, nor can I be discriminated against for that. On-call is not an essential function of being a software developer, with very few exceptions—all of which have nothing to do with "policy" or "fairness".
Needless to say, companies (and some coworkers) really don't like this.
I had a co-worker at a previous job, he did two or three on-call rotation and told our boss that he couldn't do it. Mentally it's simply to much for him, especially outside business hours where he felt alone with to much responsibility. In terms of abilities and qualifications he was absolutely able to do the job. Nobody complained or got angry with him over it, because everyone could relate.
At the other end of the scale I had another co-worker, in a more complicated scenario who absolutely didn't care. The payment for the on-call shifts was very good, so he just grabbed as many as possible. He would just take his laptop golfing, no problem. His reasoning: Either he'd know how to fix the problem, if not he'd just to call someone else and hand of the incident.
Imagine being a single parent; or needing to support elderly or disabled family.
That said, I'd hope most places would be willing to accommodate you. Places I've worked have always treated oncall as a kinda optional "right thing to do". I've never seen anyone punished for missing an alert. You'd have a good argument if that were the case at your company but that approach to oncall is not universal.
For the more common scenario of on-call rotation, it would be very difficult to make that argument because other people can take up the disabled person's shifts.
Product should be on oncall rotation, too, even if it is as a shadow. This is an important feedback mechanism about the choices they make.
And there are plenty of cases where you need to fix something ASAP and the Ops team can't do it.
That last point was very nice as it meant you could work on whatever you felt was most important for quality of life improvements all week long while not fielding on call issues. This meant that I looked forward to on call.
I think it is also why less and less people having stuff on premise and going for SaaS/Cloud solutions.
If someone wants to have my software on their servers and me not having any access better they have dedicated person for running it and I don't care even if they pay $1000 per hour - I am still not dealing with server I don't know talking on the phone with admin that has no clue how my software should be configured.
Am I the only one that thinks the prospect being on call one week every six is horrifying?