Deadman Check

How a deadman check works

A deadman check is built around missing signals, not positive thresholds. Instead of asking whether a metric is too high or too low, it asks whether a heartbeat, callback, or periodic report has arrived on time. That makes it useful for systems where silence itself is the failure signal, such as stalled jobs, hung workers, frozen telemetry, or dead integrations.

The pattern is simple but powerful: an expected event must recur within a defined window. If the window closes without a signal, the monitoring system treats that absence as actionable evidence. In practice, the mechanism is often implemented with timers, scheduled probes, watchdogs, or synthetic callbacks that expire unless refreshed.

Why absence detection matters in operations

Deadman checks are especially useful when normal health metrics can look deceptively normal during failure. A service may stop processing work while CPU, memory, or disk usage stay within range. A queue consumer may be alive but no longer draining messages. A telemetry pipeline may quietly stop exporting data, leaving dashboards blind unless the missing heartbeat is detected directly.

This makes the pattern a good fit for monitoring critical liveness conditions, especially where time-based guarantees matter more than resource saturation. It is also valuable for distributed systems, where partial failure can be harder to spot than a full outage. The signal is not “something is wrong” in a generic sense, but “the thing that should be happening has stopped happening.”

Common implementations and signal sources

Deadman checks can be implemented at several layers. Application code may emit a regular heartbeat event. Infrastructure schedulers may expect a job to re-register itself. Observability platforms may watch for a stream of metrics or logs to remain continuous. Some teams use synthetic transactions or callback acknowledgements to confirm that a workflow is still progressing.

The key design choice is selecting a signal that truly represents progress or liveness. If the signal is too coarse, the check produces false reassurance. If it is too noisy or too frequent, it becomes expensive and hard to trust. Good implementations pair the heartbeat interval with an alerting window that reflects the system’s normal timing variance, restart behaviour, and tolerance for brief interruption.

What makes deadman checks useful as a detection pattern

A deadman check is not a full monitoring strategy on its own, but it fills an important gap in detection. It catches silent failures, stalled automation, and data path breakdowns that threshold-based alerting can miss. It also works well as a last-line safety signal for scheduled tasks, backups, report generation, and any workflow where successful completion is usually confirmed by continued signal flow.

In mature monitoring designs, deadman checks complement logs, metrics, and traces rather than replacing them. They are most effective when the alert is tied to a clear owner and an unambiguous operational response, so the lack of signal becomes a specific incident to investigate rather than a vague warning that gets ignored.

Risk and Threat Considerations

Silent failure is the main risk with deadman checks, because the very condition they are meant to detect can also hide evidence from operators. If the heartbeat source, scheduler, or transport path fails, the system may lose visibility into stalled processing, failed backups, or broken telemetry exactly when it needs that visibility most.

Failure mechanism: The expected signal stops because the job, service, or reporting path has stalled, been misconfigured, or been interrupted, and the absence alert becomes the only indicator that something is no longer progressing.

Impact: Delayed detection can extend outage duration, allow failed automation to persist unnoticed, and leave downstream systems acting on stale assumptions or missing data.

Practitioner Guidance

Why practitioners should care: Deadman checks are most valuable when liveness matters more than nominal service status. Use them for workflows where a missing heartbeat is itself the business risk, such as unattended batch processing, backup completion, or telemetry continuity.

Common misunderstanding: Teams sometimes treat a deadman alert as a generic uptime check. It is more specific than that, because it monitors continued activity over time, not just whether a service answers once. The threshold and window should match the normal cadence of the signal, or the alert will be noisy and ignored.

Practitioner takeaway: Anchor the check to a signal that reflects real progress, then make sure someone owns the response when that signal goes quiet.

Related resources from NHI Mgmt Group

• Why do attackers often check model availability before trying to generate content?
• What should security teams check before using chat to build provisioning workflows?
• What should organisations check before rolling out zero standing privilege at scale?
• What should organisations check before standardising on adaptive MFA?