Show HN: I built an open-source tool to make on-call suck less (github.com)
I am building an open source platform to make on-call better and less stressful for engineers. We are building a tool that can silence alerts and help with debugging and root cause analysis. We also want to automate tedious parts of being on-call (running runbooks manually, answering questions on Slack, dealing with Pagerduty). Here is a quick video of how it works: https://youtu.be/m_K9Dq1kZDw
I hated being on-call for a couple of reasons:
* Alert volume: The number of alerts kept increasing over time. It was hard to maintain existing alerts. This would lead to a lot of noisy and unactionable alerts. I have lost count of the number of times I got woken up by alert that auto-resolved 5 minutes later.
* Debugging: Debugging an alert or a customer support ticket would need me to gain context on a service that I might not have worked on before. These companies used many observability tools that would make debugging challenging. There are always a time pressure to resolve issues quickly.
There were some more tangential issues that used to take up a lot of on-call time
* Support: Answering questions from other teams. A lot of times these questions were repetitive and have been answered before.
* Dealing with PagerDuty: These tools are hard to use. e.g. It was hard to schedule an override in PD or do holiday schedules.
I am building an on-call tool that is Slack-native since that has become the de-facto tool for on-call engineers.
We heard from a lot of engineers that maintaining good alert hygiene is a challenge.
To start off, Opslane integrates with Datadog and can classify alerts as actionable or noisy.
We analyze your alert history across various signals:
1. Alert frequency
2. How quickly the alerts have resolved in the past
3. Alert priority
4. Alert response history
Our classification is conservative and it can be tuned as teams get more confidence in the predictions. We want to make sure that you aren't accidentally missing a critical alert.
Additionally, we generate a weekly report based on all your alerts to give you a picture of your overall alert hygiene.
What’s next?
1. Building more integrations (Prometheus, Splunk, Sentry, PagerDuty) to continue making on-call quality of life better
2. Help make debugging and root cause analysis easier.
3. Runbook automation
We’re still pretty early in development and we want to make on-call quality of life better. Any feedback would be much appreciated!
173 comments
[ 4.1 ms ] story [ 196 ms ] threadI guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphical dashboard to learn the rest of problems
What strategies have you seen work well?
I have no idea what's the best setup for small companies
But, if alarms are configured in a clean way, ideally your team is getting some warnings and such there and then if there's an alert that needs to actually page, it sends that to PagerDuty or whatever platform you use along with another message to Slack.
We had a rule in my team (before a management change that blew it all to shit) that we don’t use email or messaging for monitoring. Everything goes into the SOR. Once it’s in the SOR, if people want emails, texts, or whatever, it let them know there is work to do, that’s up to the team. Others would make dashboards… lots of options once it’s in the system, and nothing gets lost.
For example, I went from a team that looked at tickets all day to one that mainly worked on user stories in Jira. Because no one was looking at the incidents in the SOR, things were getting missed. I wrote something to check for incident tickets assigned to our team every hour, and it would post them in our team chat so people knew there was work to do. Then once per day, it would post everything still unassigned, so if something was lost on that hourly post, it would annoy everyone once per day until it was assigned/resolved. It worked out decently well. If there was a lot of stuff, it would post a message to have someone actually login to the SOR and look at all our tickets. I would sometimes use the standup to assign stuff out and get some attention on it, if things were getting bad.
We are working on a variant being used more by investigative teams than IT ops - so think IR, fraud, misinfo, etc - which has similarities but also domain differences. If of interest to someone with an operational infosec background (hunt, IR, secops) , and esp US-based, the Louie.AI team is hiring an SE + principal here.
Anyone know of a similar alert UI for data/business alarms (eg installs dropping WoW, crashes spiking DoD, etc)?
Something that feeds of Snowflake/BigQuery, but with a similar nice UI so that you can quickly see false positives and silence them.
The tools I’ve used so far (mostly in-house built) have all ended in a spammy slack channel that no one ever checks anymore.
In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used.
Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here.
Regardless, well done for making something.
However, the design is pretty flexible and we don't want to tie ourselves to a single platform either.
Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken).
Technical tools cannot fix culture problems!
edit: management not talking to engineers, or being aware of problems and deciding not to prioritize fixing them, are both culture problems. The way you fix culture problems, as someone who is not in management, is to either turn your brain off and accept that life is imperfect (i.e. fix yourself instead of the root cause), or to find a different job (i.e. if the culture problem is so bad that it's leading to burnout). In any event, cultural problems cannot be solved with technical tools.
However, one of the things that I noticed in my previous companies was that my management chain wasn't even aware that the problem was this bad.
We also wanted to add better reporting (like the alert analytics) so that people have more visibility into the state of alerts + on-call load on engineers.
What strategies have worked well for you when it comes to management prioritizing these problems?
Isn't that a cultural problem?
"We're paying down our technical debt"
The existing tools mentioned can show the metrics. Management needs an education - and that is part of the engineering job.
Isn’t that bizarre? In all my years as an engineer I can count the number of managers that went to learn about engineering by themselves, on one hand.
It’s literally their job, but somehow they feel they can do it without understanding it.
I think engineer brings the numbers to management to decide course.
I prefer the situation where the CTO has no MBA and worked their way up - but that is uncommon IME.
So, in many orgs, engineer puts their comms hat on an presents a solid case.
The engineer who can communicate well, and show the metrics is typically the one who can get promoted to the decision maker role. First from the bottom up, then as a great leader
That part is fine. What I do not understand is why there is so little interest in learning what makes engineering different from running a widgets factory.
“Tell me why it won’t work” is a fine question, but it’d be nice if I didn’t have to force all their education on them.
E.g. how many managers ignore that oft repeated adage that 9 women cannot have a baby in a month, and just spam more people on a project in the hope it’ll go faster.
The problem are not the alerts — the alerts actually are catching real problems — the problem is the following:
1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prioritized 3. A culture on the team which allows alerts to go untriaged due to desensitization.
Our headcount got reduced by ~40% and — surprise surprise — reliability and on-call got much worse. Senior leadership has made the decision that the cost cuts are worth the decreased reliability so nothing is going to change.
The job market is rough so people put up with this for now.
I don’t mean to imply there is some kind of failure magnitude competition, I just want to reinforce that software “engineering” already has a huge problem with abject neglect of the learnings that other sign-and-stamp engineering fields have already learned from and fixed. Us code slingers are not in uncharted territory, we just need to learn from our predecessors and peers that build literal bridges and towers and force management to treat our field in the same way.
In my experience, it’s not a net win. They’ve budgeted the same amount for compensation either way, so you’re probably getting lower base comp if they’re allocating some of it for on-call.
It also creates an atmosphere where on-call becomes more normalized, because you’re getting paid extra to do it. Some people, usually young single people, will try to milk the overtime for as much as they can, dragging out the hours spent doing on-call work because every extra hour spent on the problem makes their paycheck bigger.
The number of people who started pushing code with subtle bugs so they could create a ticket for it, fix their own bug, and get closer to that $100 gift card was shocking to me.
I can’t imagine the chaos that would occur if something came with a $10K bonus attached. Some people will bend over backward to get even tiny rewards. Dangling a $10K reward would get the wheels turning in their heads immediately.
I was lucky enough to join a company where management does this. The managers were made to do this by experienced engineers who explained to them in no uncertain terms that stuff was broken and nothing was being shipped until things stopped being broken. Unless you have good managers this won’t happen without a fight and it’s a fight I think we as engineers need to take.
Some managers in other teams played the “oh it’s not super high impact it’s not prioritized” game, and those teams now own a bunch of broken stuff and make very slow progress because their developers are tiptoeing around broken glass, and end up building even more broken stuff because nothing they own is robust. Those managers played themselves.
Communication with management is bidirectional, sometimes they need a lot of persuasion.
Sounds like managing up, i.e. doing IC workload and the manager's job. Hard pass.
I doubt very much that my experience was unique. In my new position we have the same problems with reliability but I don’t get involved in the political side of trying to argue about it, just turn up and do my 9-5. I’m a lot less stressed now!
That said, I like that you're 'saying out loud' with this. Slack and other similar comm tooling has always been advertised as a productivity booster due to their 'async' nature. Nobody actually believes this anymore and coupling it with the oncall notifications really closes the lid on that thing.
We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications.
We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that alerts me on configurable notifications on my phone. Carrying this dongle for the week would be a physical reminder I'm on call.
Does anyone know anything?
I have no affiliation with OpsGenie outside of using at work.
And we have one user with another brand that also locks down the notification/alert settings and kills apps in attempt "to save battery" which can't be controlled.
You can already enable silence/focus time bypass modes for apps like PagerDuty and such…
If you can’t develop some sense of responsibility to check if you’re on-call, frankly you have no business being in an on-call role.
No hardware will make you or your engineers more diligent. The only reason pagers made more sense than phones is/was because of protocol reasons NOT because it’s some separate device.
I realize it's a "me" problem and therefore I'm looking for a solution. Others in the company have the same problem. That said: This is my very own company and I have a great sense of responsibility but I also have a shit-ton of other things in my head and I'm not the only one with this issue here.
The silence etc. bypass doesn't always work (I commented in another thread).
grimace face
I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and we can certainly use more competition in this space, but this aspect of it is basically enabling bad cultural practices and I wouldn’t highlight it as a main selling point.
This way there is data to make a case that certain alerts are noisy (for various reasons) and we should strive to reduce the time spent dealing with these alerts. Fixing some of them might be as easy as deleting them but for others might need dedicated time working on them.
However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers.
I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.
You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as such, and more context without using brain power is always welcome. I think I'll give it a shot.
Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.
Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.
Anyway in lot of places management see IT as a necessary evil. Like a service center similar to HR but less popular because management genuinely don’t understand it and most IT departments lack HRs political shrewdness and communication abilities. At the same time it’s not uncommon for users to be unable to tell support if they’re on an Android or iOS device (yes, I’m serious). Sometimes employees won’t even differentiate between their professional and personal IT issues on their work devices. Which means that sometimes they’ll raise hell over things that might not warrant full alert systems for on-site support.
What might be challenging here is that you’ll still need someone to actually use the authors tool correctly. Though that is probably going to be a lot easier than making any sort of change management to was organisations relationship with IT.
In similar mindset, I've seen attempts to "fix" flaky test suites by retrying failing tests 5 times until they pass. What happens: You just set a new baseline of shit allowed. This allows even more noise to enter the system and you have to rerun them even more or need increasingly more advanced tools to filter out the noise.
Once the new baseline is anchored, you become dependent on the filter. Now every tool that interacts with the metrics need to be aware of the additional filter, there may be more than just the slack messages. Should your dashboard show the raw or the filtered metrics?
Devils advocate: consider an alert with 99/100 false positives. The LLM may be good at classifying it as noisy but will it do a better job than a human to react to the 1 true positive? Maybe, but at the same time it allows more such noise to accumulate in the system, in effect a net negative. It's better to remove such an alert instead. Even if the numbers were turned around in favor, that's a lot of complexity added.
The additional context this product provides may of course still be useful and i applaud the effort. This product space does have a lot of potential for growth and is a real pain for operators. Be careful with using it as a substitute for proper alert hygiene and culture.
I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have unintended consequences. Imagine a developer telling management that they didn’t implement error checking on some edge case because the support team told them they didn’t have documentation about how to take action for instance. The appropriate response there would be “why on earth are you asking support permission to add error messages for a known error?”. On the other hand, if a support team is drowning in noisy error messages they need tooling to make it easy to distinguish between those and other messages that need to be reviewed of have action taken.
That's true, but if the error says "PANIC! EVERYTHING IS DOWN" when it's not true, then it's asking for an action that's outsized to the problem. Error messages are fine, but they just need to be classified and responded to correctly, and noisy alerts are typically the ones that are misclassified and demanding attention they (probably) don't deserve.
If the error is not-actionable, why wake someone up in the middle of the night because of it?
I don't think anyone is rejecting the observability of these errors, but just that there's no point in having it alert/wake someone up unnecessarily.
The unintended consequence of forcing alerts down ops' throat is them gradually caring less about pages, because there's a very good chance that each one is unactionable. I've worked places that do this, I've seen it happen first-hand more than once.
It starts with frustration and ops being less helpful to devs, and ends in a jaded acceptance where ops people start telling each other "just close it and see if it happens a second or third time, that alert never means anything". At that point, the system may as well not emit the errors anyways because no one is looking at the alerts anyways.
My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes.
The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.
If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring.
So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.
Consider a not-that-hypothetical example: "host computer is unreachable" alerts that page oncall when they arrive for members of a fleet of critical database servers or replicas.
The alerts have proven their usefulness (they tend to arrive several minutes before application-level error spikes when a database is e.g. so overloaded the monitoring agent can't function or a replica is gone so changelogs are overflowing) ... when they're genuine. However, they're mostly not genuine: alerting agents crash and automatic-restart-service init scripts bug out or give up; per-database-owner customizations in hosts' available file descriptor numbers are propagated incorrectly to non-database services and prevent the alerting agent from running, databases that serve infrequent-but-critical on-demand reporting loads are subjected to tens-of-minutes-long load spikes during which the host is doing what it's supposed to but so pegged that the alerting agent won't work, and so on.
What do you do with those alerts?
"Just fix the problems causing the false positives!" Fine, but that takes a lot of time and coordinated effort, even if the oncall folks are empowered to prioritize the work getting done (which is far from a given at many companies, for reasons both good and bad): auto-restart-agent scripts can be replaced with better scripts (time, effort, debugging of some hokey bash that needs to run on a wide variety of environments) or systemd (time, effort, maintenance windows, and approval/retraining to update ancient linux distributions running critical databases). File descriptor/per-database tunings can be unified and continually audited for/invalidated before configs are pushed (developer effort, coordination with teams writing configs). Reporting databases can be upsized (money, maitenance windows) or the database processes can be moved into a cgroup to leave some resources to spare (effort, distro upgrades, maintenance windows).
That's going to take awhile, if it ever happens to completion.
Meanwhile, this "host unreachable" alert is useless 90% of the time and very useful (as in: it can be leveraged to prevent downtime for customers entirely) the remaining 10% of the time.
Like, sure, some of those issues are stupid. But none are hypothetical, all are younger than 5y, and I bet this kind of struggle is common and representative even at companies who are invested in operations and operations staff.
That's not an "inability to create useful observability", that's a genuinely hard problem resulting in noisy, spurious alerts that, depending on the rate-of-change/regulatory space of the company, might persist for months or years. What's more, alert management is an ongoing process. Even if one family of noisy alerts is addressed, another one will emerge as new behaviors and technologies are adopted.
I guess this is all to say that I don't think tools like Opslane (which I have not used) are "enabling bad cultural practices". Organizations that don't give a shit about operations will continue to suck at operations no matter what tools they use. But products Opslane are valuable even (especially?) in capable, operations-focused organizations as well.
https://betterstack.com/community/guides/incident-management...
Also helpful is having management that is responsive to bad on-call situations and recognizes when capable, full-time around-the-clock staffing is really needed. It seems too few well-paid tech VPs understand what a 7-Eleven management trainee does, i.e., you shouldn't rely on 1st shift workers to handle all the problems that pop up on 2nd and 3rd shift!
> Opslane is a tool that helps (make) the on-call experience less stressful.
I'm also working on an open source incident management platform called Incidental (https://github.com/incidentalhq/incidental), slightly orthogonal to what you're doing, and it's great to see others addressing these on-call challenges.
Our tech stacks are quite similar too - I'm also using Python 3, FastAPI!
You don’t need that Times Square ad, only 8-10 people will look up. If you just want the footage of your conspicuous consumotion, you can easily photoshop it for decades already.
Similarly, chat causes anxiety and lack of productivity. Threaded forums like HN are better. Having a system to prevent problems and the rare emergency is better than having everyone glued to their phones 24/7. And frankly, threads keep information better localized AND give people a chance to THINK about the response and iterate before posting in a hurry. When producers of content take their time, this creates efficiencies for EVERY INTERACTION WITH that content later, and effects downstream. (eg my caps lock gaffe above, I wont go back and fix it, will jjst keesp typing 111!1!!!)
Anyway people, so now we come to today’s culture. Growing up I had people call and wish happy birthday. Then they posted it on FB. Then FB automated the wishes so you just press a button. Then people automated the thanks by pressing likes. And you can probably make a bot to automate that. What once was a thoughtful gesture has become commoditized with bots talking to bots.
Similar things occurred with resumes and job applications etc.
So I say, you want to know my feedback? Add an AI agent that replies back with basic assurances and questions to whoever “summoned you”, have the AI fill out a form, and send you that. The equivalent of front-line call center workers asking “Have you tried turning it on and off again” and “I understand it doesn’t work, but how can we replicate it.”
That repetitive stuff should he done by AI and build up an FAQ Knowledge Base for bozos and then only bother you if it came across a novel problem it hasn’t solved yet, like an emergency because, say, there’s a windows BSOD spreading and systems don’t boot up. Make the AI do triage and tell the differencd.
Did you search tools that are cheaper than DD?
I once worked for a string of businesses that would just send everything to on call unless engineers threatened to quit. Promised automated late night customer sign ups? Haven't actually invested in the website so that it can do that? Just make the on call engineer do it. Too lazy to hire off shore L1 technical support? Just send residential internet support calls to the the On Call engineer! Sell a service that doesn't work in the rain? Just send the on call guy to site every time it rains so he can reconfirm yes, the service sucks. Basic usability questions that could have been resolved during business hours? Does your contract say 24/7 support? Damn, guess thats going to On Call.
Shit even in contracting gigs where I have agreed to be "On Call" for severity 1 emergencies, small business owners will send you things like service turn ups or slow speed issues.
But I would'nt use LLM for it due to hallucinations
You don't have to be a missile defense system to require a stable system where devs respond quickly...
Odd, because I'm not sure if you are either.
Very few things are so important or costly, and if you're winding up in that situation frequently enough to rob people of their personal time to manually deal with it, clearly something is majorly wrong with this hypothetical critical thing at an architectural level.
There's nothing controversial about this.
Any outage that costs more than paying an engineer to be on-call is worth it. It's not that complex. If an outage blocks 10000 people from doing their work, it's obviously worth it to wake someone up to try to resolve it half a day sooner. (Someone you've been paying specifically for this purpose!)
> rob people of their personal time
Being paid to be on-call is not your personal time.
Just fix the original problem, don't layer an LLM into it.
We wanted to provide that awareness because a lot of teams arent fully aware how bad the problem might be (on-calls change weekly, there might be a bunch of other issues)
The only way to make on-call less stressful is to do the boring work of preparing for incidents, and the boring work of cleaning up after incidents. No magic software will do it for you.
We're building a tool in the same space but opted out of using LLMs. We've received a lot of positive feedback from our users who explicitly didn't want critical alerts to be dependent on a possibly opaque LLM. While I understand that some teams might choose to go this route, I agree with some commentators here that AI can help with symptoms but doesn't address the root cause, which is often poor observability and processes.
Granted, neural networks were not generally applicable to this problem at the time, but this whole idea seems like the same problem being solved again.
Telecoms and IT used to supervise their networks using Alarms, in either a Network Management System (NMS) or something more ad-hoc like Nagios. There, you got structured alarms over a network, like SNMP traps, that got stored as records in a database. It’s fairly easy to program filters using simple counting or more complex heuristics against a database.
Now, for some reason, alerting has shifted to Slack. Naturally since the data is now unstructured text, the solution involves an LLM! You build complexity into the filtering solution because you have an alarm infrastructure that’s too simple.
If it were an alarm database, an operator would be able to 1. Acknowledge the alarm 2. Manually clear an alarm that was issued in error.
Without those mechanisms, alarm handling becomes really difficult for an ops team, because now all you have is either a string of emails or a chat log.
Meanwhile they either have a global workforce so don't actually need to have anyone on-call or only have customers in countries they have employees in.
It's bullshit.
Either companies should be up front about this when hiring or it should be optional and paid.
Or, they can use their engineering talent, just like telecoms companies have been doing for ever, to engineer their products to be more resilient and automate failure cases so proper remediation can wait until working hours.
good to see more options in this space! especially OS. I think de-noising is a good feature given alert fatigue is one of the repeating complaints of on-callers.