A cluster can appear healthy because individual pods are running and basic status checks are green, while an underlying dependency has become unstable. In the example described, etcd could not reliably write to disk because of storage latency, which then broke SkyDNS and application connectivity. This shows why health monitoring must test real service behavior, not just process presence.
Kubernetes health checks can miss dependency failure
Kubernetes can report green status when the control plane and individual pods are still responsive, even though a deeper dependency is already degrading service. That mismatch usually appears when liveness or readiness signals are limited to process presence instead of validating the storage, DNS, or network path the workload actually needs to serve traffic.
This is why a cluster can look healthy during an outage that users can already feel. The system may still schedule pods, report node readiness, and pass superficial checks while the real failure sits in an upstream or downstream dependency that those checks never exercise.
A practical takeaway is that “healthy” in Kubernetes often means “the platform can still observe the workload,” not “the workload can still complete its job end to end.”
Why etcd, DNS, and storage can fail without changing the overall status
In the failure pattern described on the page, etcd becomes the hidden dependency that starts the cascade. If it cannot write reliably to disk because storage latency is too high, cluster state updates slow down or stall, and services that depend on timely state propagation begin to fail even while surface-level health indicators remain normal. For a cluster operator, the important point is that control plane responsiveness and application availability are not the same thing.
DNS is often where the problem becomes visible first. When SkyDNS or another cluster DNS component depends on a backing service that is struggling, name resolution can degrade before pod restarts or node alerts make the issue obvious. That is why storage, state management, and service discovery should be treated as part of the application path, not as invisible infrastructure.
The operational lesson is to trace the dependency chain from workload to storage and back again. If the chain breaks anywhere, Kubernetes may still continue to report partial health because the broken component has not yet crossed the threshold that the platform uses to mark it down.
Why real service checks are more reliable than process checks
Health monitoring becomes useful only when it tests the user journey or a close proxy for it. A process can be alive, a container can respond to a probe, and the cluster can still be failing if the application cannot reach its data store, resolve names, or complete a transaction. That means readiness probes, synthetic transactions, and dependency-aware checks provide much better signal than simple “is the process running” tests.
For clusters with stateful control components, the best check is usually one that exercises the thing most likely to break under load, contention, or latency. If the weak point is storage, then probe storage behavior. If the weak point is DNS, validate name resolution. If the weak point is state propagation, confirm that the cluster can actually commit and observe state changes, not just expose a health endpoint.
Healthy-looking clusters often fail in the gap between observability and usability. The more important the service, the less acceptable it is to rely on a status signal that does not validate the dependency the service actually needs.
Risk and Threat Considerations
A kubernetes cluster that appears healthy while a core dependency is degrading creates a dangerous false sense of stability. The main risk is delayed detection, because operators may continue to trust green checks after user traffic has already started failing or timing out.
Failure mechanism: A dependency such as storage, DNS, or state storage slows down or becomes unavailable, but the probes watching pods or nodes do not exercise the failing path, so the cluster remains marked healthy while real service delivery degrades.
Impact: Recovery is delayed, troubleshooting starts in the wrong place, and cascading outages can spread because the underlying fault is not isolated early.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-53 Rev 5, CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | SI-4 — System Monitoring | Health checks and dependency failures require effective monitoring of actual service behavior. |
| CA-7 — Continuous Monitoring | The question is about continuous validation of cluster health beyond superficial status. | |
| Recommendation — Monitor service behavior and dependency health, not just process presence. Continuously validate the service path, including storage and DNS dependencies. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Operational visibility is needed to detect dependency failures that health checks can miss. |
| Recommendation — Log dependency errors and alert on degraded service behavior early. | ||
| NIST CSF 2.0 | DE.CM-01 — Networks and network services are monitored to detect potential cybersecurity events | DNS and service-path degradation need monitoring that goes beyond pod status. |
| DE.CM-08 — Vulnerability scans are performed | The failure pattern involves latent infrastructure weakness that should be surfaced through ongoing assessment. | |
| Recommendation — Monitor networked service dependencies and alert on path degradation. Assess infrastructure dependencies that can silently undermine service availability. | ||
Practitioner Guidance
What to verify: Validate that your health model includes at least one check for the most brittle dependency in the service path, not just pod uptime. If a workload depends on storage latency, DNS resolution, or state writes, the probe should prove that those functions still work under normal and degraded conditions.
Decision rule: If a green status does not answer “can the service still complete its real task?”, treat that status as insufficient and add a deeper probe before using it for operational decisions or automation.
Practitioner takeaway: The useful question is not whether Kubernetes can still observe the workload, but whether the workload can still serve correctly when its weakest dependency is under stress.
Related resources from NHI Mgmt Group
- What are the signs that SOC detection is failing even when dashboards look healthy?
- Why do Kubernetes workloads need runtime controls even after cluster configuration and image admission are already in place?
- What are the signs that a Kubernetes workload is failing even when the pod still looks healthy?
- Why does configuration drift create compliance risk even when controls look healthy?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 26, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
