How does a deadman check differ from a threshold alert?

What each pattern is telling you

A deadman check is about continuity: it answers, “is the expected signal still arriving?” That makes it useful when absence is itself a failure mode, such as a missing heartbeat, stalled job, stopped exporter, or broken telemetry path. A threshold alert is about magnitude: it answers, “has the measured value crossed a boundary?”

That difference changes how you interpret the alert. Deadman checks are usually binary and time-sensitive, while threshold alerts are often comparative and trend-aware. One is designed to detect silence quickly; the other is designed to detect abnormal behaviour before or after it becomes operationally significant.

In practice, a deadman check should be tied to something that ought to emit regularly enough for a missed signal to mean something. A threshold alert should be tied to a metric that is stable enough to make the limit meaningful, otherwise you create noise instead of detection.

When to use one instead of the other

Use a deadman check when loss of a recurring event is the problem you care about most. Common examples include missed heartbeats, a paused integration, an agent that stopped reporting, or a pipeline that has gone quiet when it should still be working.

Use a threshold alert when the system can still be active but unhealthy. Examples include CPU saturation, elevated latency, error spikes, queue depth growth, or unusual request volume. A system can breach a threshold while still sending healthy-looking telemetry, which is why thresholding does not replace a deadman check.

The two often complement each other. A service can be both alive and degraded, or completely silent because it crashed, lost connectivity, or stopped publishing metrics. Good monitoring separates those cases so responders know whether to look for overload, failure, or missing telemetry.

Why the distinction matters operationally

The main failure risk with deadman checks is false silence. If the expected signal is too sparse, delayed, batched, or intermittently blocked, you may page on an issue that is actually just a timing problem. The main failure risk with threshold alerts is false normalcy, where a metric is still reporting but has drifted far enough to hide a real service problem until user impact is visible.

Thresholds also depend on good baselines. A static CPU or latency limit can be too blunt for spiky workloads, seasonal traffic, or systems that have known bursts. Deadman checks depend on dependable emission paths, so the alerting channel, collector, and upstream job cadence all become part of the control.

Blockchain Analytics and On-Chain Monitoring Guide is a useful illustration of this distinction because monitoring systems often need both heartbeat-style checks and threshold-style thresholds to separate silence from abnormal movement.

Risk and Threat Considerations

Silence can hide compromise just as easily as it can hide a fault. If an attacker disables a collector, suppresses telemetry, or disrupts the monitoring path, a deadman check may be the first signal that visibility has been degraded. Threshold alerts, by contrast, can be evaded when adversaries operate below a trigger level or spread activity across time to avoid a sharp spike.

Failure mechanism: The deadman control fails when the expected heartbeat is not reliable enough to distinguish normal gaps from genuine loss of visibility, while thresholding fails when the limit is too coarse, too noisy, or too easy to stay just under.

Impact: Teams can miss either a full stop or a slow-burn incident, which means response starts later and investigation has less trustworthy telemetry to work with.

Practitioner Guidance

What to verify: Make sure the deadman signal is emitted on a schedule tight enough that a missed interval is meaningful, and confirm that the alerting path itself is covered by the same monitoring design. For thresholds, verify that the chosen limit reflects real operating behaviour, not a convenient default copied from another service.

Decision rule: If the question is “did it stop reporting?”, choose a deadman check first. If the question is “is it operating outside the acceptable range?”, choose a threshold alert first. In mature monitoring, you usually need both, because loss of telemetry and abnormal values are different failure classes.

Practitioner takeaway: Treat deadman checks as visibility controls and threshold alerts as state controls, then tune each one to the failure mode you actually need to detect.

Related resources from NHI Mgmt Group

• How do organisations decide when to use a GitOps check instead of a later alert and ticket process?
• Alert severity threshold
• Spend Threshold Alert
• Alert Threshold