Ask HN: Who else is working/on call over Christmas?

162 points by deathgripsss ↗ HN
I volunteered this year to be the on call over Christmas and currently I'm dealing with a generator fire. Merry Christmas.

Who else is working through the 25th this year?

102 comments

[ 0.24 ms ] story [ 60.6 ms ] thread
I'm not working but oncall. Fingers crossed. I always volunteer (I did a few years in a row) as usually it's quiet (no deployments, people understand it's a holiday) and it's a small price to pay to not feel bad for always trying to sneak out of being oncall for new year's eve :). .

Best of luck OP and merry Christmas too!

On-call here, but it's mostly because it arrives on a weekend, all I'm required to do is Acknowledge Alerts that are usually false positives, unless some big service disruption occurs
if the alerts are usually false positives you need new alerts!
Yes, but not so simple. Especially when the alerts are caused by 2 systems down the line and aren't owned by our immediate Team. That and the fact that we're migrating to a better thought out architecture which has been 2 years in the making. Slow pace at big corps..
you definitely shouldn’t be on call for alerts managed by another team. a lot of smells in the observability story here, i’d be pushing for an overhaul
I’m not on call, but I’m really happy that the guys from home alarm provider are on call. We had a maltunction and they promptly reacted. Cool - and the merry evening could go on. Merry christmas.
Own a little SaaS. Gave everyone the week off between Christmas and New Years. Have been happy to do this 5/6 years we have been in business. Love giving it to my team for all the hard work they do all year.

Machine wise k8s is rock solid and I never need to touch it.

Client wise, will be a few support tickets…

And hey, we don’t have to rely on anyone else to work over Xmas.
> Gave everyone the week off between Christmas and New Years.

Do you mean additional to their PTO? That's nice, many people feel obliged to take it off for family even if they'd rather do something else with a whole week off and see family on weekends and such.

Yes, kind of a forced vacation week. I think people come back in Jan feeling less burned out… vs having to sneak some random work days in on Dec 28th or whatever.
That’s a nice gesture; hopefully you can keep it up!

I once worked at a place that tried to always give close the office two extra days during that span. As long as Christmas wasn’t a Wednesday, they always made it such that you had an extra long weekend twice in a row, and announced it in early November so people could plan well enough in advance.

They also, for planning purposes, assumed that there were only 11 months in a year. That is, each person was only expected to do one full month of work total in Nov and Dec. it gave people time to catch up, work on tech debt, learn something, experiment with something, shop, take vacation, etc. I’d say that more often than not it was overall more productive.

It's mildly common in the US moreso in the blue collar side of things, but I've noticed some tech companies that do it too. Mine does.

It's not like anyone gets anything done.

Ever since I’ve been in charge of my first team, this has been something I believed strongly in. One of the highlights of this holiday season for me was finding out that the new lead who took over one of my teams is doing the same thing.

Out of curiosity, what were the circumstances that made this not workable for the remaining 1 of the 6 years?

Ah, I was like 2 weeks post my 3rd baby with the others still pretty small. No way I could leave Mom alone!

But awesome enough someone from the support team stepped up and took it for the team so everyone else got the week off! So it’s cool to have a cool team.

Good on you for giving folks extra time off.
Very nice of you. And I second k8s's robustness. Have a little one person startup with 100k users. Went on 5 weeks vacation in October and November and never had to worry about a thing. Everything kept going and just had to answer a few support mails here and there.
on call, in case someone has a major cyber incident. so far, so quiet
Currently putting the finishing touches on a proposal that is, for some unfathomable reason, due the 28th of December.
This is my wife, who has to turn in a research paper on 31st.

On the flip side, I get to spend Christmas helping to make sure that related academic code doesn't succumb to any stereotypes ... :-)

Running a decentralised network. Some nodes not behaving well.
I am doing a python freelance job. It sucks.
Sorry to hear about your generator fire. Power stuff is usually a major drag because most of us on the IT side can’t do anything about it.

I’m on-call ops support for a public safety answering point (a “911 call-center”) this weekend. It’s a regular call rotation and my number was up for this one. It’s usually quiet— except when it isn’t.

I’m gonna go on-site to a Customer’s on-prem server room tomorrow morning and swap out a bad drive in a SAN. I’m pretty happy because I know there’s no chance of a “Hey, while you’re here-” event! Empty offices mean no unexpected questions!

I usually volunteer, whether or not I actually end up working varies.

Fingers crossed for this year. I'm hopeful, we've done well to improve reliability and coach the junior teams

Working on my own startup so putting my all into it.
I am always on-call. Pages that fall through, end up paging me. Happy Christmas, New Years everybody and stay safe!
Best of luck with the generator.

I'm on a hotline, volunteering, for people who might be feeling extra mental stress due to the holidays and might do something they regret. I decided not to risk the travel to see family with the storms and I figured I might as well try to help others who might be dealing with some of the same tough emotions that often plague me this time of year.

Thank you! Have a Merry Christmas.
The world always need people like you. Thank you.
you're doing something good. here's a virtual hug.
You are doing a ton of good this Christmas. Thank you for your efforts here and there.
I'm on call this past week and next week!

My on-call is a bit odd because it's both as OSS maintainer deciding whether I need to coordinate a hotfix release, as well as certain functionality within a managed service (PaaS) enabled by the OSS project. Only the latter thing can actually page me, and I don't anticipate being paged. Fingers crossed!

I was surprised to find that my startup's contract dev kept working on the 23rd and even 24th. If he's pinging me with questions and sending me new versions to test, I'm going to answer and test stuff out! It's not mission-critical though (it's an manifest v3 transition, which Google has given a reprieve on), so we'll take a few days before getting back to it.

The good thing about testing browser extensions is that you can do much of it while just surfing the web! Cheers to everyone else who are clocking in during the holidays!

Meeee (OnCall). But my team is small/I don't really have to do anything (fingers crossed). I do dislike that my phone volume is on, when normally it's off, so every notification is making noise

Press 4 to acknowledge.

You can make an exception for the pager duty providers number. That will it will always make a sound no matter what.
Thanks for the tip
On call, but this is the best time of the year to be on call: none of my coworkers are around to break anything. A gut estimate of my pager load is that 80 percent of pages are caused by bad changes pushed by humans, and another 10 percent are caused by changes pushed by humans exercising latent errors in the system, and only the remaining 10 percent random shit breaking.

The main downside is that I also don't have as many people to escalate / help when an outage does occur over the holidays. But that's not as urgent now that say, generator fires are Amazon's problem not mine.

“Generator fires are highly correlated with Christmas weather” is the best sales pitch I have heard for cloud services.
This time of year, you do get the pages for things that were always broken, but nobody noticed before, because they only show up when the system has been running without changes for more than two weeks.
This actually happened to us last week in fact.

No deployments revealed how a legacy background processor started losing connections to the message queue and gets stuck in a state where it never reconnects.

Deployments always cycled the pods before the issue manifested.

This is something a (now former) colleague of mine pointed out: that the kubernetes descheduler can enforce a maximum lifetime[0] that sort of forces continual reboots. So if your system cannot tolerate running for a long time continously, this is one method to gracefully restart long running pods.

[0]: https://github.com/kubernetes-sigs/descheduler#podlifetime

christmas is the busiest time for one of my customers, and just last week one of the data servers had hardware issues that ended up in everything but the disks being replaced because the hosting provider could not find which component was actually broken. i am busy rebuilding raid since, but already 2 out of 3 raid devices restored. it was touch and go for a while, with a risk of loosing the whole server (because raid kept claiming that disks were faulty when they really weren't) i am already rethinking the architecture to make it more robust and am trying to convince the customer to invest in new servers for that.

i am the only one in my role, so i'll be monitoring the rebuild process over the holidays. that said, i don't celebrate christmas anyways, so in a team i'd be the first to volunteer to be on call at this time to allow others to relax, while i'd be taking time off on my own holidays.

(comment deleted)
Just had a pointless alarm go off that resolved after a few minutes. According to the Runbook there isn’t even any action that can be taken!
nonactionable alert? thats a paddlin
I am a volunteer at a research lab right now in another state over my winter break. Had the opportunity to take a break, but I don't know anyone here and won't be here for a long time so thought what the heck
I am marking exams. My deadline to enter the final grades is December 27th.
I'm on call 24/7. One of the downsides of a small company.

I go months between needing to do anything though.