Ask HN: Options for back end engineer sick of oncall
I'm close to hitting a decade of full time experience. In that time, I have always worked in back end engineering roles requiring oncall duties. I've recently identified a few things that are my key stressors at work, and the key contributors to my burn out. The top of that list is the existence of oncall duties, and the need to be ready to respond for a week at a time. Even with a quiet oncall, I still plan my personal life around being oncall. Same goes for planning any tie off. I've reached a point where I don't need or want to self sacrifice (e.g,. for total comp, or prestige, or whatever); instead, I'm trying to tweak my path forward for the next decade or so of work.
I'm not ruling out a change of disciplines, for instance mobile or frontend development. That said, I have a strong set of skills and deep experience from the last decade. So I'm wondering, what sorts of roles might be less of a skill reset than what I generalize as "back end" engineering?
Some background on me, if it helps:
* Management is doable but not enjoyable (I dislike the context switching, miss building things, and can stomach process as an IC but don't enjoy driving process).
* Years of experience in security engineering
* Some enjoyment of using basic data science (ETLs, Jupyter notebooks analyzing warehoused data, etc) to inform eng decisions in the past
* Some low level Linux programming experience for about a year or two, but pretty surface level ("understands procfs")
* Of course, the rest is all just building CRUD apps hitting a variety of database types and generally doing so in products with millions of customers, where downtime or breaking changes are not tolerated. Pretty vanilla stuff. Distributed systems to the extent of "distributed CRUD" (lol)
When times are good, I'm very happy being an IC engineer. I can handle the demands of engineering work and the organizational challenges that come with it, when contained to a reasonable business day. I'm just worried that I can't sustain what I've been doing for so long (exacerbated by being oncall quite regularly in my current role) and don't want to sacrifice my personal well being (or family's) any more. I'd rather figure out a new role than do something drastic like quitting out of burn out, only to start again in another backend role that basically has the same problems as my current situation.
Thanks for reading!
31 comments
[ 4.8 ms ] story [ 82.8 ms ] threadI currently support three web apps for different customers, and I will get called if something blows up. I have monitoring on the apps and cloud resources so I usually find out about problems before my customers or their users do. I have implemented deployment processes to prevent breaking changes as much as possible, and rollback/recovery processes just in case. I get two or three emergency calls a year.
I have had jobs in the past with managers who told me I would have to do on-call duty. I told them no, my kids and personal life mean more to me. I agreed to carry a phone or pager, and get to problems as fast as I could, but not let on-call intrude so much I was effectively working extra hours for free. I never got penalized or fired for setting some limits. I never knew a manager or executive who was on call 24/7 or carried their laptop around at the golf course.
- 1 week in-call every 5 to 6 weeks (depending on how many engineers are available for on call)
- if you get paged, you need to acknowledge within 10-15 min. Sometimes it’s nothing and you can go back to sleep, but sometimes you may need your laptop to fix things. The point is: you don’t know before hand
- it’s paid. In my opinion the money is not worth it
- it’s mandatory. But even if it’s not, if all your teammates are on call and you’re the only one who refuses it, well, that usually creates friction
When I see this kind of thing with my customers it comes from a combination of unrealistic expectations ("We need 99.9999% uptime for our marketing sites") and over-engineered and fragile infrastructure, code, and development/deployment processes ("Our K8S/microservices architecture can fail in many ways that no one understands.")
I did some work for a company a few years ago that had a cron job that rebooted the web server every day, because that was the solution they had come up with to "fix" the server crashing (Apache + Rails in that case). The actual problem was running MySQL and Apache on the same server, leading to MySQL eventually starving Apache for memory. Once I fixed that Apache + Rails would run for years without needed a reboot. I had another customer with a server that ran out of disk space every couple of months, requiring frantic resizing of an attached disk. Moved them to S3, problem solved, no more calls.
"I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone."
It's possible we're working on different kinds (or scales) of software or differently sized organizations – there are no human customers actually "calling," nor can you typically resolve true production incidents (let alone root cause them or estimate impact) from a phone.
(Yes)
In the ancient times, when you had separate roles for programmers, testers, system administrators, etc., all working in the waterfall environment, there was no need for a programmer to be on call. The programmers developed during their working hours. When the product was ready, it was tested by the testers. When it finally got to production, it was mostly bug-free. You had the system administrators on call, in case some server caught fire.
After the testers were mostly eliminated, bugs in production became more frequent. Continuous deployment into production requires continuous on-call duty. Devops means that the programmers are now also responsible for the failures of servers. Now the programmers are expected to also be agile in their free time.
(But of course, we are collectively all too smart and too proud to form a union. I guess we have to suffer the obvious consequences then.)
The more fellow programmers i see, the more im convinced it’s not enough for such a demanding job.
You have to personally care to some degree about your users, about the general problem you’re solving, you need to feel meaning in what you do.
Then oncalls etc also start show some meaning and value to the bigger picture.
Without it — you ll face plenty of energy sinks in any discipline.
I’m in the same situation as OP, and it seems they and I, we are just normal professionals who care about their career, but are not willing to sacrifice our scarce free time in exchange for more money or whatever.
I think, in the end, it's a bit of both, you need to care about the product to actually do it, but you are also correct in that the tradeoff has to be worth it.
First it’s not a theory but observations. Many of them.
Second, i dont see how your argument refutes the theory if it was a theory?
Managers may and in fact are usually on call from what I personally have seen. They just have different type of oncall within their areas of competence, and it’s often bat called “oncall”.
I personally think quite the opposite: the better you take care of your work hygiene the better you are capable of serving your users.
But if you don’t care about your jobs goals at all - no WLB tricks will last long for you.
Nope. Of course I don't. It's just a job. I am paid to deliver value during a pre-agreed set of hours and nothing more. I will do my work to the best of my ability during that time, after which I dngaf. The company, in turn, will treat me as a non-entity when it suits (eg. will lay me off in hard times). It is for the owner of the company/stakeholders to care about the mission, since they are the ones who profit from that care (and from the naive care of others who have no stake and yet are tricked into one-sided "loyalty"). Life is too short to put that much care and effort into making someone else a ton of money.
This is another signal supporting my thesis.
I am personally convinced that as soon as your job requires a complex and hard activities from your brain - it inevitably touches all its cicruits responsable for our deepest motivations, values etc.
It doesn’t mean you have to be crazy about your job etc, but it means a part of your personality is working towards some common goal too, not just your skills.
And personality tends to be very picky about how it wastes its relatively short lifetime.
It's possible to believe in a company mission, and even to drink the company Koolaid to the extent one might romanticize on YN (I did in my early days, while also resentful of the demands of oncall), but it is not necessary to give all the hours of your day to any company. Also, I do enjoy engineering, so your hypothesis is also off base in that regard (in fact I don't really care much for the idea of professionalism). I know plenty of accomplished and happy people for whom work is just one part of their lives and not their defining attribute.
I left DevOps/infra for infosec/risk mgmt in part due to on call rotations, and would never accept a job again that required them.
Doesn't sound so bad as I write it and remember it... :)
I've worked for a dozen different companies (contractor) over the past 10 years as a backend engineer and have never been faced with oncall. Neither have my permanent employee colleagues. In many cases we had no direct access to production environments as that was managed by separate teams. In the one startup I worked at, it was so early stage that there were only ever a half dozen users, and the CTO handled any issues that popped up over the weekend.
The type of companies that had dedicated teams for maintaining live environments included big banks, government departments, insurance companies, large healthcare companies.
That's interesting, so if some bug pops up during the weekend they will just revert back? And then that bug gets handled later. Or is there also so rigorous QA process that bugs rarely happen, and it's more like infra issues that the "product support" team can handle on their own?
The interesting part is that I think on-call is not only about infra issues etc., but could also be bugs in the application or some other weird behavior, which I'm assuming the "production support" wouldn't know how to handle. Maybe there is some huge amount of process to handle all that I'm just oblivious of, never having heard of this before?