My team has a meeting to hand off the on-call to the next person, and we discuss all pages we got during the week. Primarily two things: whether the page was for a good reason or not (good: our on-call person had an something actionable to fix. bad: non-actionable pages, pages because someone else's system was broken, false alarms, etc), and also whether there is something we can do so we never get paged for this again. I find it very effective at reducing pages.
Lol yeah. My old team had oncall pages in the middle of the night pretty often where nothing was actually the matter. My manager was only nominally on call. In the handoff meetings every week he was basically just like “that sucks”.
I never understood why companies didn't simply leverage 24x7 internet MSPs.
They are able to staff 24x7 by spreading the cost over multiple customers and working through the process of making your application manageable by a 3rd party is super beneficial.
Most of these companies will also do performance monitoring and analysis as well.
They see issues and optimization opportunities across multiple applications and know more than a single team who's only built one.
That works well for generic IT systems and running the desktop/laptop fleets, but doesn’t work at all for running the software a company builds.
We typically split our teams, so we have ~16 split across two time zones so that our shifts are just 12 hours during the day. It works well, but it is expensive, so we support a lot of services (or a small number of very high priority services) as a result.
I hadn't heard of Managed Service Providers before, but you make a good case for them.
I'm finding surprisingly little discussion on HN regarding the costs/benefits of MSPs. Or rather, under which conditions (such as company size) they make sense.
I have occasionally convinced teams to adopt both oncall and sprint cycles aligned with Tuesday [1] - the dev teams all loved it. Management was a harder sell, but by and large were happier with the extra days to communicate results/get metrics before their own Friday deadlines.
[1] also Wednesday/Thursdays. Wednesdays were my favorite in good working environments, it felt like running a successful marathon, but it was more prone to falling apart due to short-term thinking.
I’ve always been partial for Friday night through Friday night.
You start off over the weekend, when you have energy and can survive the two days alone. Ideally no Friday releases so the transition is calm, but as the writer says the batches might fail.
You spend the week fixing whatever breaks. You’re cleanly off the Monday to Monday sprint, just doing on-call/ops.
You finish Friday evening and immediately get Friday night and the weekend to recover when you need it most.
That was exactly my reasoning too when I set up our on call roster as Friday to Friday, though for us Saturday is the busiest day in terms of customer activity, so it was a no-brainer.
This post was discussed somewhere else and I saw someone say that their work does Firday noon to Friday noon and their work gives the outgoing on call the rest of Friday off. I feel like that's even better because 1) it recognizes the hard work that the outgoing on call put in 2) it give the incoming on call a few hours to get up to speed while they still have the support of the other engineers on the team.
Maybe I'm taking you too literally, but I wouldn't want to have a handoff sync-up (or any meeting, really) on a Friday night, nor push that earlier so significant things can happen between sync-up and the actual shift in responsibility from person to person. Friday-to-Friday does sound good.
One thing I really liked in a previous job was a split daytime-vs-nighttime rotation. It was well worth a little annoyance to set up in our tools. One week you'd be the 'daytime' oncall for business hours (something like 9-5 Mon-Fri, though we might have tweaked those hours a bit; it might have been 10-6 or something). The next you'd be on call for the complementary time (5-9, weekends). You were on call for the same total amount of time, just smeared over two different weeks. It ended up being less of a burden to optimize your schedule for a reasonable response time, but operational work still got done. And in practice awareness of operational issues was not too hard to maintain between the two members of the split.
(I think the best thing, if you can swing it, is probably a follow-the-sun rotation where there are three teams distributed 8 hours apart around the globe, and they trade off 8-hour workday shifts. But a lot of uncommon things probably have to be true of your organization for that idea to even be on the radar.)
We split in-hours and out-of-hours, and I wouldn't want it any other way. It's especially good if a late night incident keeps you from sleep, because you can take the time back the next morning and let someone else pick up the pieces :).
My team does Wednesday to Wednesday for many of the same reasons mentioned in the article, and it works great. We switch at 11am and hold a hand-off meeting at that time, and invite the whole team.
Hand-off meetings with the whole team work really well (in my opinion!) when you have a relatively small team--we have 9 FT teammates. Often someone else may have been delegated the page or bug that arose and can discuss how they handled it, or someone who wasn't involved may have insight for how to handle a situation better the next time. Since we're all going to be on rotation at least once during a quarter, it's great to know what happened in case a similar page pops up later.
Finally, we also fill out a running Doc before/during the meeting with links to the pages/bugs, along with short descriptions of how they were handled. This forms a great living memory of how to deal with incidents, and is also often the birthplace of new playbooks for handling new types of incidents.
Same here. Except we do a two week rotation, and it aligns with our sprints. The active on-call engineer doesn’t have any assigned sprint work and focuses their effort on fixing bugs or cleaning up the backlog when they’re not actively triaging an incident.
At Google we used to get paid an oncall bonus which was calculated at something like 1/3 your prorated salary for the non-working hours you were oncall (IIRC), up to some limit per quarter. For my team a week of oncall per quarter would max it out and net you a few thousand dollars bonus.
On my teams, if someone got paged off-hours they would just work less the day after the event. imo it should just be part of the regular salary/work expectations, incentivizing keeping oncall low
I've wished for a tech workers union for this reason. I don't care about pay, let the union say nothing about pay.
But let's align incentives. Any time spent fixing issues on-call is compensated 4-to-1. Workers may accrue compensation time, and any compensation time in excess of 20 hours is paid 10-to-1 when the employee leaves. The idea here isn't for workers to accrue and cash out comp time, but instead to give an incentive to the organization to ensure workers use their comp time.
Let's align incentives, what's hard on the worker should be hard on the owners and management.
All you need is three people that agree on one realistic change at your workplace and you have a union. From there you start having regular meetings and plan a strategy for pushing this single issue.
When that's done, chill for a while, do some recruiting and education in the workplace and think about what the next realistic change ought to be.
Gross, no. This just allows management to ignore problems and push development teams to do feature work, even when everything is on fire and the oncall person is getting paged multiple times per day.
Oncall should be compensated, always. The oncall person should get a flat rate just for being on standby, and should also receive a per-page payout, and that amount should be larger if the page happens outside regular business hours.
Then management will actually realize there's a cost to pushing features and pulling in deadlines at the expense of robust engineering practices. Or they can decide they are fine with that, and paying the oncall person is a cost of doing business they way they want to.
I've seen too many instances either issues they come up during oncall never get fixed, and just page and page and page.
I will never again work at a company where oncall is "just a part of the job". I value my own time too much.
> Or they can decide they are fine with that, and paying the oncall person is a cost of doing business they way they want to.
I was going to say, this would almost certainly be the outcome. Companies have no problem throwing millions at AWS, DataDog, etc. They certainly aren’t going to blink at an employee making a couple hundred bucks extra per day.
In France it is mandatory either with salary or rest. In addition the labor code stipulates mandatory daily rest of 11 contiguous hours even during the weekend and extra 24 contiguous hours of rest during the weekend.
Hours of intervention are considered work.
In my company we get approximately 800€ for every week of on-call and each hour of intervention is also compensated with salary.
From my point of view this should be high enough for the company to be willing to focus on on-call issues.
Ater years of being on-call I must admit the salary is comfortable but it doesn't cover the pain and constraints of being on-call: being kinda "stuck" at home basically, lots of consequences on private life etc.
I was on-call on a Saturday at the start of November and the prod issue took nearly 10 hours. No extra compensation from my company (not required by law in my country for saturdays but come on!). A week later I had to do it Monday night again, still no extra compensation... I can work less with these extra hours but... when?
In my previous job, the company followed the country's laws pretty much to the letter. Simply being on call gave some small fraction of my "hourly rate" (despite being a full time employee), which actually does add up, since e.g. on a weekend you are accumulating 48 hours of such on call time (while on weekdays it's only 16 per day, as your actual 8 hours worked is not counted twice).
If there was an actual incident, you'd get paid as if that was overtime worked, and it depended on when it occurred (e.g. weekends and public holidays carried a higher than normal multiplier). There were also limits on how much rest you'd need to be guaranteed, etc.
On average, our actual incidents were relatively infrequent, and the pay out mostly depended on the size of the team, which dictated how often you got rotated in. It worked out to something like +10% salary though.
In a previous job yes. 7200DKK per week for on-call. Hours where between 17:00 and 08:00 on weekdays and all of Saturday and Sunday, running Friday to Friday. From 8:00 to 17:00 the normal service desk would handle incidents.
If you got paged you'd get 150% of your hourly pay, per started hour. So if you got pages at 22:00 and again at 3:00 that's three hours of pay, regardless of each issue only taking 5 or 10 minutes to fix.
That's roughly $1000/€950 per week of on-call, plus the hours. You'd have four/five of these per year and you could pick up an extra month of pay per year with the standby pay, plus the hours and maybe pick up another day here and there.
Holidays where normally distributed on a volunteer basis (you'd still get paid, but you opted-in to those days). So maybe I'd be home on New Years, but out on Christmas, so I'd offer to cover Christmas, while another colleague might care more about being able to drink on New Years.
Originally we where almost 50 people handling on-call, so you'd have one week per year, but that's not sustainable, you forget how to handle common issues or how to fill out incident reports and handoff correctly.
The most stupid on-call schedule I ever did was midnight to midnight, every other day... It was only me an my boss. That was incredibly stupid, because you couldn't go out on one day, and the day where you could go out, you had to be careful about having one drink to many.
At my last job I got time off in lieu for actual hours worked + about $4/hour for being on-call.
I really wish we'd gotten paid for hours worked rather than TOIL, not for personal preference but because it would have aligned the company's incentives better. We might actually have fixed some of the problems if not doing so cost the business a tangible sum of money.
That's the way it was 20-30 years ago. If you were on call, it was informally understood that you were going to be rolling in late in the morning, at if something big happened, you'd be missing a day or two afterwards.
Worked well until E&Y came in and "fixed" things with a strategic plan.
Yes, a flat daily fee during the week, and double that during the weekend or public holidays. Comes to ~600€ per week. If you actually get paged, you automatically get time off for the amount of hours you spent dealing with the incident.
At mine: compensated for being on-call, but it's a pittance (in the 100-200 per week range, which is nowhere near being worth it). We get TOIL for time spent responding to incidents.
Last job I was compensated a flat $450 for being on call 1 week, on top of my salary in the 140K range.
Everybody received the same on-call pay and I'm pretty sure salaries were similar but not identical.
Google paid SREs 67% their hourly rate for tier 1 oncall outside business hours, regardless of whether they were paged. So 12h shifts on weekends were a full day's pay. Convertible to Off-in-Lieu. I had so many off days. Not sure if it's still the case after those layoffs.
Not that way around, no -- paying extra when paged creates a perverse incentive. We're paid to make ourselves available, and encouraged to take any time we actually spend working out of hours back in lieu.
My team have put a lot of effort into only rarely being paged: a normal week on-call won't have any out-of-hours activity at all.
We're being compensated for being available (more on weekends and holidays) and get additional compensation for every 30 minutes incident response. Two incidents in one night? Twice the money. Additionally, we are required to work less the next day (we're also required by law to have at least eight hours of free time between two work days. So if you get an incident seven hours after you got off work? Congrats, you now have to wait eight hours before you can start working. This is of course very annoying, so nobody does that)
I'm not even sure if doing on-call duty without compensation is legal in my country.
In the past, there were some cases of "fake" incidents, but the amount of documentation makes sure that the company is able to crack down on this.
We are on-call for 48hrs at a time, about once every 12 days or so, one day as backup, and one as primary. It's nice because it doesn't interrupt your week too much. The downside being that complex issues might require extra work while not on-call
We do Thursday to Thursday and then you get Friday off after completed on-call.
Being on-call gives you no extra pay by itself, but if you get paged off hours and need to work you get paid 150 to 200% of your normal hourly wage depending on what time of day you need to work.
No pay for being on call by itself is still poor, particularly when it comes to swapping rotations between team members to provide flexibility amongst each other.
You’re making yourself available 24/7. That has a non trivial lifestyle impact which I’ve always thought deserves more than is typically rewarded.
Not to mention that there is incentive to keep having oncall pages, because that's how you get paid. Or not participate at all. On the other hand, with a flat payment, there is a big incentive to prevent issues and not have(reduce) ooh incidents, and participate in the rota.
As long as the on-call coverage is as specified at the time of hiring, this is just a difference in form of payment.
If I receive 100 total units of compensation, I'd way rather get 100 units of base pay (and 0 on-call pay) than 90 units of base pay and 10 units of specific on-call pay. (What if the company eliminates on-call? What if I get injured and my insurance only covers base pay? Severance is usually based only on base pay; I would not be paid on-call while I'm on PTO or other paid leave, annual raise percentages typically apply to base pay, etc...)
How can the on-call coverage be specified at hiring? Can the company guarantee that my team will never shrink or that the page rate won't increase?
What will financially encourage my company to stop paging me overnight if there isn't a labor cost to the company every time an on-call incident occurs?
> What if I get injured and my insurance only covers base pay?
Insurance payouts can be easily based on wages that include reported commissions, tips, and overtime. They can very easily be based on an average of past actual wages paid in the last handful of months at the company.
> Severance is usually based only on base pay
Severance is a completely optional practice that is based entirely on what the company wants to do. I would argue that severance is more accurately based on "The lowest safe number to pay to this particular employee to make sure their termination does not become a legal risk."
> I would not be paid on-call while I'm on PTO or other paid leave
But also, PTO days and on-call days don't indersect. If you took time off during an on-call shift you would be trading it with a team member, so you would never lose that extra wage.
Example: I'm taking a week off, it's during my scheduled on-call shift. I would normally get paid my on-call hours but I didn't this week. But when I get back from my vacation, I'm picking up an extra on-call shift because my team member covered my shift when I was on vacation.
Now, I'm taking a week off, but it's not during my on-call shift. I wouldn't have been paid on-call hours this week anyway. When I get back from my vacation, I am going on my normally scheduled on-call shift.
I personally have never felt compensated dynamically enough for on-call schedules. Most corporate jobs seem to pay for a sliver of the life disruption, maybe paying for half my phone and Internet bill or something like that. They all say that the on-call is baked into the compensation, but I'm not so sure.
> If you took time off during an on-call shift you would be trading it with a team member, so you would never lose that extra wage.
I think this is true in _most cases_, but is not a given. I myself have encountered scenarios where it isn’t true: switching with someone much later in the rotation, only to then end up having to switch again for instance. You could envision a nefarious teammate weaseling out of their fair share with sneaky switches like this, too, though paying for it would maybe incentivize them not to!
Germany (among other countries) has laws around this. My company pays I think 200 euro a day that someone is on call, so my German reports end up making a decent amount in months they have their on call shifts, especially felt when the team is smaller and rotations more frequent!
>"Severance is a completely optional practice that is based entirely on what the company wants to do. I would argue that severance is more accurately based on "The lowest safe number to pay to this particular employee to make sure their termination does not become a legal risk."
Almost right! I see it as an extension of what I call the basic rules, "I am as nice to you as you are to me", and "I care exactly as much as you do."
That does, in some cases, expand severance a little beyond the cold risk calculation. If the severance is going to someone who helped the company make it, then helping make sure they make it to their next gig is part of the equation.
Not everyone boils it all down that far, but a whole lot of us do!
Which makes your comment solid, and mine a quibble, but one I consider worthy of some discussion.
+1! You can't travel very much, you can't go hiking or biking in places without cell coverage, your whatever thing you are busy with gets interrupted, you can get woken up in the middle of the night, etc etc. That deserves some compensation.
In OP's case it sounds like they do get compensated with the day off, which is PTO. It's not a trade everyone would make but an extra day off into a long weekend is one I would have taken earlier in my career.
The extra day off is probably equivalent to getting paid ? Last time I had on-call part of the job, I think the pay increase for standby would have amounted to roughly 8h as well (actual interventions were also 150% for regular nights and Saturday, 200% for Saturday night and Sunday)
Lugging around a laptop and the on-call phone when going anywhere, checking every now and then when the phone was not with your for a while (e.g. pool, gym etc), making sure you don't go places with no signal was enough of a PITA that knowing we were paid every hour of that had a nice psychological effect.
We do daily shifts with a follow the sun rotation, makes it easier to handle persistent commitments and ensures a bad week doesn't all land on the same person.
In some cases it might help. Because then it becomes “natural” - on-call thing. It’s not something someone dreads as in “god, that week is coming”. Also, it spreads the fuck-ups and peaceful times better.
Increasing hand-offs by 7x is sub-optimal and interferes with folks wanting to take continuous vacation time. Again, I can see for ops teams that could be true, but very much disagree that on-call should be a "natural" thing for dev teams in the first place. It can be a necessary evil that should be minimized (the personality of people who like firefighting and quiet development are quite distinct and there are people who actually like the former.) On the latter point, I think that benefit is very much a mirage. If there's a flaw in the system causing a "bad week," it actually might be easier for the first person who gets the hang of it to deal with it than to try handing off and teaching the next one in the rotation.
This! Whenever someone talk about on-call this aspect of that rotation gets swept under the carpet. Whenever I interview I always ask whether they have on-call system (they must if there are servers and apps involved) and if they do whether they have follow the sun.
Most don’t even like the question. For them such questions are red flags or the candidate is not “motivated enough”. Rarely some even have follow the sun policy. They might have one in their HQ, true for a lot of US/EU firms, but their offices in a developing country like India - it’s always something on the lines of “oh, engineers here take full ownership; they are the owners”.
Also, I have seen — 2-3 days rotation with follow the sun is best, week long or longer being worst.
Then there are companies where it could be forever on-call with no follow the sun - e.g. Amazon, Uber (in India at least). That’s another world altogether.
I mean it sounds clever, but how do you have engineers with no expertise in a system handling calls for it? I've been at places that follow the sun, and you frequently have no idea what to do during an incident, because the person who owns the system is offline. But at least you can sit one a useless incident call during work hours instead of completing your own tickets I suppose.
There should be SOPs in place for each "expected" issue so people know what to do.
Its not like you (should) start debugging and deploying stuff in the middle of your on-call shift anyway.
Its not 100% for sure but in the normal case this should be fine.
The point is that the "ownership" used to rhetorically justify the labor of such engineers is not "ownership" at all: it's missing the literal most important aspect, that is the ability to profit proportionally as a project brings in revenue. Literally everything else is included -- the intensity of work, the singular focus, the care and devotion, the expected level of initiative -- but not the part that would most benefit the engineer.
Not only that but if you have a multiple continental team then no one needs to be waken by an emergency meow. (My PagerDuty is set to a meow sound. So we practice meow driven development: I don't want to hear my phone meowing piteously.) Say, you have someone on the US west coast they can do 10am-10pm while someone else in continental Europe being nine hours ahead can do 7am-7pm.
It used to be somewhat common for websites/services to go down for a few minutes every so often for maintenance/migrations/etc.
Tech entrepreneurs should give no weight to this. The market seems to support engineers doing on-call rotations, and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing.
Tech entrepreneurs should stop giving excessive “nines” of availability. Even 99% is probably enough for most customers to never notice, and significantly easier to engineer than 99.999….
The issue is less giving out excessive SLAs - it’s more that even a tiny ass startup thinks they need high availability and four nines and scale to billions - when in reality almost no one actually needs it.
But those are cool engineering problem so we’d rather work on them than on building a business.
> and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing.
The connection makes sense but one must not think in this order. One must think “people will pay for this” and then consider “does this need to be highly available?”
If you have more than one road to choose from, and one of them doesn’t require high availability, then give that one some bonus points for that.
It's still common. Nobody really cares if a site is offline for a few minutes. You try again later (or not, but so what). Heck, nobody cares if they are offline for half the day, it gets fixed and at the end of it it's just a post-mortem for the nerds to read and a shrug and life goes on for everyone else. People vastly overestimate the importance of anything that is on the public internet. None of it is life-critical (if it is, it certainly should not depend on an internet connection or a web server being up).
Correct. Things are breaking all the time. If you’re not a hospital or air traffic control, nobody is going to die if your website goes down.
There’s a time and a place for heroics, but we go to it for shit that doesn’t really matter, or worse, allow the culture of heroics to cover up the real problems that are much harder to fix.
Imagine Walmart telling all their customers "Come back tomorrow, maybe, we can't manage to keep any of our stores open. At least nobody is going to die."
An internal service we relied on when working at Norway's biggest government agency had opening hours. If you called it outside 8-17 it wouldn't reply, heh.
Government is a good one. Healthcare is another. Anything geography-locked.
I’m not saying no one should create a highly available web service. I am saying that this is one of those things that techies assume, and shouldn’t, because it’s a huge plus to hiring, business and engineering simplification, and morale if you can define away non-business-hour problems.
I agree it's worth considering, but allowing problems on e.g. weekends sets up weird incentives for management everywhere I've been.
For example, they may not want to fix quality issues as long as their consequences can be pushed to the weekend. Or they may start to demand people work weekends to do maintenance.
Or -- worst of all -- they realise they can avoid deployments entirely on weekdays, and then do these big bang deployments on weekends.
This makes engineer's lives miserable but looks like rational optimisation to management.
Original reason for this schedule was that on-call was paid by days per quarter in a tiered system so this guaranteed that all members got the 5% on-call for 10 days/quarter rather than one person hitting 9 days and dropping to 3%, but I stand by this as a better on-call rotation.
The number of people does need to be not wholly divisible so the days rotate so if you run into this you can combine Fri into Sat/Sun or break Sat/Sun apart. It’s a bit complex to set up but the mental impact of on-call is greatly reduced and if you need a week for vacation you can much more easily find someone to cover your shift for a couple days in a nearby week rather than ending up with 2 weeks back to back 6 weeks from now. And if you pull a weekend you get the week off rather than losing your weekend to on-call and going into a work week still on-call.
Any company that makes it an employee's responsibility to find "someone to cover" their on call time while they're on vacation is a company worth quitting.
I'm pretty sure that'd be illegal here in .au
On call coverage while an employee is on vacation is a management problem, not an employee problem.
Could not agree more, any company I've worked at with an on-call rotation has always ensured that staff are not scheduled when they have holiday booked. The only time an employee needed to find their own cover is if something unexpected came up during their on-call period and they needed a few hours out (like an emergency visit to the doctor with a child etc).
At my current job we have an automated scheduler which uses our gcal to ensure that it never schedules if people have an AFK entry. It also schedules fairly based on how long since the person was last on-call, not putting them on on a weekend if they were on last weekend etc (we do 24hr shifts).
Got it, thanks. Well, if you ever decide to open it up I'd be keen to having a look at it. I have yet to see a good automated scheduler and none of the places I've been so far had an automated solution that worked fine when it comes to holidays, scheduled time off, etc. A lot of manual work was put into scheduling shifts, which I always found very disappointing.
> The only time an employee needed to find their own cover is if something unexpected came up during their on-call period and they needed a few hours out (like an emergency visit to the doctor with a child etc).
That's exactly the time where "finding your own cover" is the most stressful.
I do see your point, but I think some context about where I work helps here. It's a very chilled company and all teams are very self-organising. In the case of me needing to find cover, I would just drop a message in our on-call slack channel and someone will almost always pick it up (and do the necessary stuff like adding an override in the opsgenie rotation). If nobody happens to see it (unlikely) then I would just let the alerts escalate to the next person who would be happy to pick it up because they know that people don't let alerts escalate without a good reason. Everyone cares about their colleagues so they want to help if they can (it's also quite a small company).
Most companies are like this in my experience. You have some default rotation going out to forever and as part of planning vacation you check whether the time you are requesting is when you're scheduled for on-call and if so ask if someone else is available to swap on-call. If you can't find someone to cover, you raise to your manager and either they'll cover it or find someone else to cover.
In my 20ish years I’ve done every possible day for oncall schedules. I would say each have pros/cons but overall I found it to be a minor difference.
Mon-Mon is nice because it’s a logical time to start something fresh at the start of the week.
Tuesday is good for the reasons in the post, Wednesday is similar.
Thursday is nice because after you’re done you can relax on Friday.
Friday-Friday is less common but can be nice because you get the satisfaction of being done on the last day of the week.
Where i work we do Friday 8AM -> Friday 8AM. We changed to that from a Monday->Monday a few years ago. Feedback has been postive. Coming off of on-call on Monday morning was just a major bummer.
So when a major issue happens over the weekend only Step 1 happens during the weekend. Step 2 involves following up with other teams, creating new alarms and updating the runbook. And all that usually happen during the week. The oncall is going to spend at minimum their Monday doing that so it’s better if the schedule reflects that."
Each person on my team has a day of the week they own, and then we have a rotation for weekends, and negotiate holiday/pto trades. I guess it really only maps correctly for a 5 person team.
We previously had a week long rotation, and some folks were initially skeptical of the idea to change, saying they were worried they'd feel like they were "oncall all the time". But, they agreed to try it for a month. That was a bit over a year ago now, and no complaints.
I think it ends up being a lower stress configuration, because it just becomes part of your normal expected work-week routine, and generally isn't as mentally draining.
It does make end of year PTO/holiday time a bit more complex to work out, but so far my team has been okay with that tradeoff.
how do you work around bank holidays? Which in some countries are almost always on the same weekday? Does the person who has Mondays just deal with not having a longer weekend like everyone else?
What about the person who has Friday? Do they never go out on a Friday evening?
Sounds a nice idea in theory but not all week days are equally inconvenient.
I agree with the sentiment, but on-call support is an unavoidable necessity given the critical nature of many systems that underpin modern society.
When we talk about on-call, we’re not referring to systems like Netflix streaming a major fight for 65 million users, but rather essential infrastructure like healthcare systems, nuclear power plants, military operations, financial markets, and the vast array of SCADA (Supervisory Control and Data Acquisition) systems that monitor and control industrial processes.
These systems are crucial to our safety, economy, and everyday lives, and downtime or failure is not an option.
Before Apple, I worked at Microsoft in the Azure team, where I logged over 2,016 hours of on-call support each year. This involved six 24/7 on-call rotations, each lasting two weeks, with responsibilities alternating between primary and secondary support. While there were certainly tough moments and challenging issues during those rotations, they also provided valuable learning experiences and helped me develop problem-solving skills under pressure.
I’ve always felt it should be split among a geographically distributed team where support hours follow the sun. It really sucks to be awake at 3am, alone, groggy and unsupported and responsible for saving the world.
If the company/product isn’t large enough to be distributed, is it really important that it have a 10 minute time-to-acknowledge?
I was once paged over thirty times in a span of 24 hours while working for a website that, in the grand scheme of things, could unilaterally improve life in the United States by shutting itself down.
Here's my on call schedule: never, and don't ask. It's my responsibility to do my job when I am scheduled, and it's management's responsibility to staff properly. If we can't agree, then we can't have a business relationship.
I can understand on-call hours if you're a literal firefighter or paramedic who saves lives. I understand that, as a building superintendent, every once in a long while you have to run out and fix a burst pipe before property is destroyed. I don't understand why some of these tech companies have on-call responsibilities like there was some hazard to life or property.
They need five nines of availability to make sure they don't lose one cent of potential ad revenue? Good luck with that, I guess, but I'll be over here actually sleeping through the night.
I’ve also done Wednesday to Wednesday, though Tuesday seems better mentally if only because there is one less day after the new week starts.
What is much better, though, is splitting the week into a 4/3 or 5/2 split, with a primary and backup on-call. Primary takes the weekdays, then switches with Backup for the weekend. You’re still sharp and aware of any current issues should the need arise, but the odds of a weekend page are (hopefully) lower, so you can relax a bit.
This of course requires enough people to have a reasonable rotation; 6 at a minimum, but 8 is better.
We started a split shift for a really busy oncall and it works out really well. It's Th->Tu, Tu->Th. So basically weekend+2 working days vs 3 working days.
Expectation is you are 100% oncall during the working day, so it works out pretty well between weekend vs non-weekend shifts.
I much prefer the shorter shifts to a full week. A full week on-call usually means delaying important project work, etc. for a full week.
229 comments
[ 3.1 ms ] story [ 245 ms ] threadLOL i wish
They are able to staff 24x7 by spreading the cost over multiple customers and working through the process of making your application manageable by a 3rd party is super beneficial.
Most of these companies will also do performance monitoring and analysis as well.
They see issues and optimization opportunities across multiple applications and know more than a single team who's only built one.
We typically split our teams, so we have ~16 split across two time zones so that our shifts are just 12 hours during the day. It works well, but it is expensive, so we support a lot of services (or a small number of very high priority services) as a result.
I'm finding surprisingly little discussion on HN regarding the costs/benefits of MSPs. Or rather, under which conditions (such as company size) they make sense.
Any big players or companies you would recommend?
[1] also Wednesday/Thursdays. Wednesdays were my favorite in good working environments, it felt like running a successful marathon, but it was more prone to falling apart due to short-term thinking.
You start off over the weekend, when you have energy and can survive the two days alone. Ideally no Friday releases so the transition is calm, but as the writer says the batches might fail.
You spend the week fixing whatever breaks. You’re cleanly off the Monday to Monday sprint, just doing on-call/ops.
You finish Friday evening and immediately get Friday night and the weekend to recover when you need it most.
One thing I really liked in a previous job was a split daytime-vs-nighttime rotation. It was well worth a little annoyance to set up in our tools. One week you'd be the 'daytime' oncall for business hours (something like 9-5 Mon-Fri, though we might have tweaked those hours a bit; it might have been 10-6 or something). The next you'd be on call for the complementary time (5-9, weekends). You were on call for the same total amount of time, just smeared over two different weeks. It ended up being less of a burden to optimize your schedule for a reasonable response time, but operational work still got done. And in practice awareness of operational issues was not too hard to maintain between the two members of the split.
(I think the best thing, if you can swing it, is probably a follow-the-sun rotation where there are three teams distributed 8 hours apart around the globe, and they trade off 8-hour workday shifts. But a lot of uncommon things probably have to be true of your organization for that idea to even be on the radar.)
Hand-off meetings with the whole team work really well (in my opinion!) when you have a relatively small team--we have 9 FT teammates. Often someone else may have been delegated the page or bug that arose and can discuss how they handled it, or someone who wasn't involved may have insight for how to handle a situation better the next time. Since we're all going to be on rotation at least once during a quarter, it's great to know what happened in case a similar page pops up later.
Finally, we also fill out a running Doc before/during the meeting with links to the pages/bugs, along with short descriptions of how they were handled. This forms a great living memory of how to deal with incidents, and is also often the birthplace of new playbooks for handling new types of incidents.
That reminds me of Amazon's abysmally bad employee discount, which was "10% off anything on the site, up to $100 / year".
But let's align incentives. Any time spent fixing issues on-call is compensated 4-to-1. Workers may accrue compensation time, and any compensation time in excess of 20 hours is paid 10-to-1 when the employee leaves. The idea here isn't for workers to accrue and cash out comp time, but instead to give an incentive to the organization to ensure workers use their comp time.
Let's align incentives, what's hard on the worker should be hard on the owners and management.
When that's done, chill for a while, do some recruiting and education in the workplace and think about what the next realistic change ought to be.
Oncall should be compensated, always. The oncall person should get a flat rate just for being on standby, and should also receive a per-page payout, and that amount should be larger if the page happens outside regular business hours.
Then management will actually realize there's a cost to pushing features and pulling in deadlines at the expense of robust engineering practices. Or they can decide they are fine with that, and paying the oncall person is a cost of doing business they way they want to.
I've seen too many instances either issues they come up during oncall never get fixed, and just page and page and page.
I will never again work at a company where oncall is "just a part of the job". I value my own time too much.
I was going to say, this would almost certainly be the outcome. Companies have no problem throwing millions at AWS, DataDog, etc. They certainly aren’t going to blink at an employee making a couple hundred bucks extra per day.
In my company we get approximately 800€ for every week of on-call and each hour of intervention is also compensated with salary.
From my point of view this should be high enough for the company to be willing to focus on on-call issues. Ater years of being on-call I must admit the salary is comfortable but it doesn't cover the pain and constraints of being on-call: being kinda "stuck" at home basically, lots of consequences on private life etc.
I would be interested in how response times are?
Mine is 15mins. So I have to respond and be in a incident call within 15mins.
If there was an actual incident, you'd get paid as if that was overtime worked, and it depended on when it occurred (e.g. weekends and public holidays carried a higher than normal multiplier). There were also limits on how much rest you'd need to be guaranteed, etc.
On average, our actual incidents were relatively infrequent, and the pay out mostly depended on the size of the team, which dictated how often you got rotated in. It worked out to something like +10% salary though.
If you got paged you'd get 150% of your hourly pay, per started hour. So if you got pages at 22:00 and again at 3:00 that's three hours of pay, regardless of each issue only taking 5 or 10 minutes to fix.
That's roughly $1000/€950 per week of on-call, plus the hours. You'd have four/five of these per year and you could pick up an extra month of pay per year with the standby pay, plus the hours and maybe pick up another day here and there.
Holidays where normally distributed on a volunteer basis (you'd still get paid, but you opted-in to those days). So maybe I'd be home on New Years, but out on Christmas, so I'd offer to cover Christmas, while another colleague might care more about being able to drink on New Years.
Originally we where almost 50 people handling on-call, so you'd have one week per year, but that's not sustainable, you forget how to handle common issues or how to fill out incident reports and handoff correctly.
The most stupid on-call schedule I ever did was midnight to midnight, every other day... It was only me an my boss. That was incredibly stupid, because you couldn't go out on one day, and the day where you could go out, you had to be careful about having one drink to many.
I really wish we'd gotten paid for hours worked rather than TOIL, not for personal preference but because it would have aligned the company's incentives better. We might actually have fixed some of the problems if not doing so cost the business a tangible sum of money.
Still, it was better than working for free.
Worked well until E&Y came in and "fixed" things with a strategic plan.
Anyway, I prefer Mon - Thu, Fri - Sun shifts.
My team have put a lot of effort into only rarely being paged: a normal week on-call won't have any out-of-hours activity at all.
I'm not even sure if doing on-call duty without compensation is legal in my country.
In the past, there were some cases of "fake" incidents, but the amount of documentation makes sure that the company is able to crack down on this.
Big ecommerce companies and even global brands like AWS and BMW require 24x7 on call without any compensation.
Best on-call I’ve had.
You’re making yourself available 24/7. That has a non trivial lifestyle impact which I’ve always thought deserves more than is typically rewarded.
If I receive 100 total units of compensation, I'd way rather get 100 units of base pay (and 0 on-call pay) than 90 units of base pay and 10 units of specific on-call pay. (What if the company eliminates on-call? What if I get injured and my insurance only covers base pay? Severance is usually based only on base pay; I would not be paid on-call while I'm on PTO or other paid leave, annual raise percentages typically apply to base pay, etc...)
What will financially encourage my company to stop paging me overnight if there isn't a labor cost to the company every time an on-call incident occurs?
> What if I get injured and my insurance only covers base pay?
Insurance payouts can be easily based on wages that include reported commissions, tips, and overtime. They can very easily be based on an average of past actual wages paid in the last handful of months at the company.
> Severance is usually based only on base pay
Severance is a completely optional practice that is based entirely on what the company wants to do. I would argue that severance is more accurately based on "The lowest safe number to pay to this particular employee to make sure their termination does not become a legal risk."
> I would not be paid on-call while I'm on PTO or other paid leave
But also, PTO days and on-call days don't indersect. If you took time off during an on-call shift you would be trading it with a team member, so you would never lose that extra wage.
Example: I'm taking a week off, it's during my scheduled on-call shift. I would normally get paid my on-call hours but I didn't this week. But when I get back from my vacation, I'm picking up an extra on-call shift because my team member covered my shift when I was on vacation.
Now, I'm taking a week off, but it's not during my on-call shift. I wouldn't have been paid on-call hours this week anyway. When I get back from my vacation, I am going on my normally scheduled on-call shift.
I personally have never felt compensated dynamically enough for on-call schedules. Most corporate jobs seem to pay for a sliver of the life disruption, maybe paying for half my phone and Internet bill or something like that. They all say that the on-call is baked into the compensation, but I'm not so sure.
I think this is true in _most cases_, but is not a given. I myself have encountered scenarios where it isn’t true: switching with someone much later in the rotation, only to then end up having to switch again for instance. You could envision a nefarious teammate weaseling out of their fair share with sneaky switches like this, too, though paying for it would maybe incentivize them not to!
Almost right! I see it as an extension of what I call the basic rules, "I am as nice to you as you are to me", and "I care exactly as much as you do."
That does, in some cases, expand severance a little beyond the cold risk calculation. If the severance is going to someone who helped the company make it, then helping make sure they make it to their next gig is part of the equation.
Not everyone boils it all down that far, but a whole lot of us do!
Which makes your comment solid, and mine a quibble, but one I consider worthy of some discussion.
If you have any national holidays, somebody still ends up being on-call for that holiday. I've been on-call for almost every US holiday this year.
Lugging around a laptop and the on-call phone when going anywhere, checking every now and then when the phone was not with your for a while (e.g. pool, gym etc), making sure you don't go places with no signal was enough of a PITA that knowing we were paid every hour of that had a nice psychological effect.
If I’m not paid to be reachable, nobody gets to complain when I don’t pick up the phone though.
Most don’t even like the question. For them such questions are red flags or the candidate is not “motivated enough”. Rarely some even have follow the sun policy. They might have one in their HQ, true for a lot of US/EU firms, but their offices in a developing country like India - it’s always something on the lines of “oh, engineers here take full ownership; they are the owners”.
Also, I have seen — 2-3 days rotation with follow the sun is best, week long or longer being worst.
Then there are companies where it could be forever on-call with no follow the sun - e.g. Amazon, Uber (in India at least). That’s another world altogether.
Cooperatives really should be more common.
It's just rhetorical trickery.
Tech entrepreneurs should give more weight to choosing markets that don’t require this
Tech entrepreneurs should give no weight to this. The market seems to support engineers doing on-call rotations, and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing.
Tech entrepreneurs should stop giving excessive “nines” of availability. Even 99% is probably enough for most customers to never notice, and significantly easier to engineer than 99.999….
The connection makes sense but one must not think in this order. One must think “people will pay for this” and then consider “does this need to be highly available?”
If you have more than one road to choose from, and one of them doesn’t require high availability, then give that one some bonus points for that.
There’s a time and a place for heroics, but we go to it for shit that doesn’t really matter, or worse, allow the culture of heroics to cover up the real problems that are much harder to fix.
Most stores close every night.
I’m not saying no one should create a highly available web service. I am saying that this is one of those things that techies assume, and shouldn’t, because it’s a huge plus to hiring, business and engineering simplification, and morale if you can define away non-business-hour problems.
For example, they may not want to fix quality issues as long as their consequences can be pushed to the weekend. Or they may start to demand people work weekends to do maintenance.
Or -- worst of all -- they realise they can avoid deployments entirely on weekdays, and then do these big bang deployments on weekends.
This makes engineer's lives miserable but looks like rational optimisation to management.
- Mon/Tue - Wed/Thu - Fri - Sat/Sun
Original reason for this schedule was that on-call was paid by days per quarter in a tiered system so this guaranteed that all members got the 5% on-call for 10 days/quarter rather than one person hitting 9 days and dropping to 3%, but I stand by this as a better on-call rotation.
The number of people does need to be not wholly divisible so the days rotate so if you run into this you can combine Fri into Sat/Sun or break Sat/Sun apart. It’s a bit complex to set up but the mental impact of on-call is greatly reduced and if you need a week for vacation you can much more easily find someone to cover your shift for a couple days in a nearby week rather than ending up with 2 weeks back to back 6 weeks from now. And if you pull a weekend you get the week off rather than losing your weekend to on-call and going into a work week still on-call.
I'm pretty sure that'd be illegal here in .au
On call coverage while an employee is on vacation is a management problem, not an employee problem.
At my current job we have an automated scheduler which uses our gcal to ensure that it never schedules if people have an AFK entry. It also schedules fairly based on how long since the person was last on-call, not putting them on on a weekend if they were on last weekend etc (we do 24hr shifts).
That's exactly the time where "finding your own cover" is the most stressful.
Mon-Mon is nice because it’s a logical time to start something fresh at the start of the week. Tuesday is good for the reasons in the post, Wednesday is similar. Thursday is nice because after you’re done you can relax on Friday. Friday-Friday is less common but can be nice because you get the satisfaction of being done on the last day of the week.
"- Step 1: handling it
- Step 2: making sure it doesn’t happen again
So when a major issue happens over the weekend only Step 1 happens during the weekend. Step 2 involves following up with other teams, creating new alarms and updating the runbook. And all that usually happen during the week. The oncall is going to spend at minimum their Monday doing that so it’s better if the schedule reflects that."
We previously had a week long rotation, and some folks were initially skeptical of the idea to change, saying they were worried they'd feel like they were "oncall all the time". But, they agreed to try it for a month. That was a bit over a year ago now, and no complaints.
I think it ends up being a lower stress configuration, because it just becomes part of your normal expected work-week routine, and generally isn't as mentally draining. It does make end of year PTO/holiday time a bit more complex to work out, but so far my team has been okay with that tradeoff.
What about the person who has Friday? Do they never go out on a Friday evening?
Sounds a nice idea in theory but not all week days are equally inconvenient.
When we talk about on-call, we’re not referring to systems like Netflix streaming a major fight for 65 million users, but rather essential infrastructure like healthcare systems, nuclear power plants, military operations, financial markets, and the vast array of SCADA (Supervisory Control and Data Acquisition) systems that monitor and control industrial processes.
These systems are crucial to our safety, economy, and everyday lives, and downtime or failure is not an option.
Before Apple, I worked at Microsoft in the Azure team, where I logged over 2,016 hours of on-call support each year. This involved six 24/7 on-call rotations, each lasting two weeks, with responsibilities alternating between primary and secondary support. While there were certainly tough moments and challenging issues during those rotations, they also provided valuable learning experiences and helped me develop problem-solving skills under pressure.
On-call support is a necessary evil.
If the company/product isn’t large enough to be distributed, is it really important that it have a 10 minute time-to-acknowledge?
I can understand on-call hours if you're a literal firefighter or paramedic who saves lives. I understand that, as a building superintendent, every once in a long while you have to run out and fix a burst pipe before property is destroyed. I don't understand why some of these tech companies have on-call responsibilities like there was some hazard to life or property.
They need five nines of availability to make sure they don't lose one cent of potential ad revenue? Good luck with that, I guess, but I'll be over here actually sleeping through the night.
What is much better, though, is splitting the week into a 4/3 or 5/2 split, with a primary and backup on-call. Primary takes the weekdays, then switches with Backup for the weekend. You’re still sharp and aware of any current issues should the need arise, but the odds of a weekend page are (hopefully) lower, so you can relax a bit.
This of course requires enough people to have a reasonable rotation; 6 at a minimum, but 8 is better.
Expectation is you are 100% oncall during the working day, so it works out pretty well between weekend vs non-weekend shifts.
I much prefer the shorter shifts to a full week. A full week on-call usually means delaying important project work, etc. for a full week.