Ask HN: How to do simple heartbeat monitoring?

With a metric, you can use a monotonic counter to serve as a heartbeat. A timestamp would work. In your monitoring system, when the heartbeat value has not increased in X minutes, you alert.

We have a decent free plan that would probably work for you.

[1] https://news.ycombinator.com/item?id=7917587

there's probably functionality built in to your other monitoring tools, or you could write a little serverless function to to it, but for me i really like to have it isolated. i wanted a tool that's not part of the rest of our monitoring stack, not part of our own codebase, and not something we manage.

Write a cron job that greps the logs and pulls on your api with curl?

With another 20 minutes of work you can remember the log size on each run and grep only what's new on the next run.

There isn’t a good way to solve this using a PUSH model that isn’t somewhat of a hack or using another external tool

Not sure about Signoz and PagerDuty, but there are plenty freemium services like UpTimeRobot that work fine for basics.

And then something like AWS CloudWatch has a lot more advanced options about how to treat missing data.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

With a lot of knobs to tune around what intervals you need and what counts as 'down', which you'll want to think about pretty carefully given what you need.

You cannot only do classic heartbeat checks but also high level API (single request and multi request) and Browser checks to make sure your service is behaving as expected.

The heartbeat process itself basically just sent pings every second on a UDP port and waited for a reply. If we didn’t get a reply in one second, it’d assume the connection was bad until a ping came again.

For example: https://blog.ediri.io/how-to-set-up-a-dead-mans-switch-in-pr...

You might want to use a different service for monitoring your stack just to make sure you have some redundancy there. Seems like you got an answer for how to do this with Signoz but if that is down then you won't know.

edit: on second read, it sounds like regular uptime monitoring for your API would do the trick

Free and easy. Not affiliated, just a happy user.

I combine it with a service like uptimerobot to get messages if the heartbeat stopped.

If you're on AWS, there's already heartbeat monitoring and that can integrate with CloudWatch to notify PagerDuty.

  #!/bin/bash

  # Add this script to cron to run at whatever duration you desire.

  # URL to be checked
  URL="https://example.com/test.php"

  # Email for alerts
  EMAIL="root@example.com"

  # Perform the HTTP request and extract the status code with 10 second timeout.
  STATUS=$(curl -o /dev/null -s -w "%{http_code}\n" --max-time 10 $URL)

  # Check if the status code is not 200
  if [ "$STATUS" -ne 200 ]; then
      # Send email alert
      echo "The URL $URL did not return a 200 status code. Status was $STATUS." | mail -s "URL Check Alert" $EMAIL

      # Instead of email, you could send a Slack/Teams/PagerDuty/Pushover/etc, etc alert, with something like:
      curl -X POST https://events.pagerduty.com/...
  fi

They usually all have pager duty integration as well.

Some examples:

Datadog: https://docs.datadoghq.com/synthetics/ Grafana cloud: https://grafana.com/grafana/plugins/grafana-synthetic-monito...