Topic starter
06/11/2026 6:37 pm
TL;DR: Manual monitoring decays as systems change faster than hand-written rules, and Sawmills' analysis shows postmortems usually fix the last incident rather than the coverage gap that allowed it. Splitting detection from paging is the real correction, because durable coverage has to follow the system as it exists now, not as it was imagined months ago.
Editorial analysis by NHI Mgmt Group, based on content published by Sawmills: “Fixing the Last Incident: Why Manual Monitoring Keeps Failing”.
Key questions
Q: How do teams stop monitoring coverage from drifting as systems change?
A: Teams stop monitoring coverage from drifting by maintaining a control inventory that maps each system to required logs, alerts and telemetry, then testing those detections after every infrastructure, application or cloud change.
Q: Why do postmortems usually improve the last fix instead of the monitoring model?
A: Postmortems often improve the last fix because teams focus on the alert or rule that failed during the incident, not the underlying monitoring model.
Q: How does a deadman check differ from a threshold alert?
A: A deadman check verifies that something expected is still happening, usually by expecting a periodic heartbeat and alerting when it stops.
Practitioner guidance
- Inventory every newly shipped component List services, endpoints, queues, scheduled jobs, and third-party dependencies deployed in the last 90 days, then compare that list to your current monitor set.
- Separate broad detection from paging Create a two-step process where anomalous findings are verified for impact and recurrence before a human gets interrupted.
- Add heartbeat checks for silent failure modes Use freshness or deadman checks for jobs, pipelines, and data flows so that missing output becomes an alert rather than a blind spot.
Bottom line: Manual monitoring fails when coverage depends on memory, because systems evolve faster than hand-written rules do.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
06/11/2026 6:42 pm
Manual monitoring decay is a control-governance problem, not just an alerting problem. The central failure is that ownership of coverage is informal, so checks age faster than the systems they watch. That creates a standing gap between operational reality and declared coverage, which is especially dangerous when the watched asset is a service account, pipeline, certificate, or other non-human dependency. NHI incident metrics gives teams a useful way to measure that drift, but the bigger point is that coverage itself must be governed as a living control.
A question worth separating out:
Q: How does a deadman check differ from a threshold alert?
A: A deadman check verifies that something expected is still happening, usually by expecting a periodic heartbeat and alerting when it stops. A threshold alert fires when a metric crosses a limit, such as CPU, latency or error rate. The first detects silence or loss of telemetry, the second detects abnormal values.
👉 Read our full editorial: Manual monitoring decays as systems change and coverage lags