A workload can appear healthy while the actual service is down if the parent process is still running but the application it launches has failed. The clearest sign is a pod staying in Running state while requests fail or return errors. Liveness and readiness probes expose this gap by checking the service itself instead of trusting only the container process state.
Why a Kubernetes workload can fail while the pod still looks healthy
In Kubernetes, “healthy” at the pod layer can be misleading because the container process can be up even when the service inside it is not actually serving traffic. A pod may remain in Running state while the app has crashed internally, deadlocked, lost a dependency, or stopped responding. The signal that matters is whether the workload can still satisfy real requests, not whether a process exists.
The first clue is a mismatch between status and behaviour: the pod is Running, but users see timeouts, 5xx responses, or missing output. That gap often appears when the container entrypoint or wrapper process survives while the actual application worker has failed. If you only watch pod phase, you miss the difference between “container alive” and “service operational.”
These failures are often exposed only when the workload is probed in a way that matches user traffic. Liveness probes test whether the workload should be restarted, while readiness probes test whether it should receive traffic. If either probe is absent, too shallow, or checks only the parent process, Kubernetes can keep routing requests to a workload that is technically running but functionally broken. For container and runtime failure patterns, NIST SP 800-190 Container Security is a useful companion reference.
What usually causes the false sense of health
A common cause is a split between process state and application state. Supervisors, shell wrappers, or sidecars may stay alive even after the real server thread, listener, or application runtime has failed. Resource exhaustion can create the same symptom: the pod is still scheduled and running, but the app cannot accept new work because it is out of memory, blocked on I/O, starved for CPU, or waiting on an upstream dependency that no longer responds.
Dependency failure is another frequent pattern. A database, cache, queue, config source, or internal API may be unavailable, so the workload looks healthy from the outside but cannot complete its function. If the application does not convert that dependency failure into an unhealthy readiness signal, Kubernetes will continue to treat the pod as eligible for traffic. This is why workload-level health checks should reflect the service contract, not just the container runtime.
For request-path validation and traffic eligibility, the SPIFFE workload identity specification is also relevant where health and trust depend on workload-to-workload attestation. If your platform uses secret-bearing integrations, the NHIMG Ultimate Guide to NHIs provides the broader identity context behind workload access and operational visibility.
Health can also degrade gradually rather than fail all at once. A pod may still answer basic checks while latency, error rates, queue depth, or connection failures reveal that it is slipping into a partial outage. In practice, the strongest indicator is often not a single probe result, but a pattern of successful container status combined with broken business behaviour.
How to tell the difference between container health and service health
The practical test is whether the workload can complete a real transaction end to end. A useful readiness check should fail if the app cannot accept or process traffic, and a useful liveness check should fail only when the process is stuck so badly that restart is the right recovery action. That separation matters because a workload can be alive, unready, and unusable at the same time.
Good diagnostics usually include three layers: process state, internal app state, and external behaviour. Process state tells you whether the container entrypoint is still running. Internal state tells you whether the application can reach what it needs to function. External behaviour tells you whether clients can actually succeed. When those three disagree, the workload is failing even though the pod object still looks normal.
At scale, this is where container health checks become a resilience control rather than a convenience feature. The workload should expose a signal that the orchestrator can act on, and operators should confirm that the signal is tied to a meaningful service dependency, not a trivial local test. Where you need a broader control reference for runtime and deployment risk, NIST SP 800-190 Container Security is the most direct external anchor.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-53 Rev 5, CIS Controls v8 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | IA-9 — Service Identification and Authentication | Covers service-to-service trust when workload health depends on authenticated service access. |
| Recommendation — Validate service trust paths before routing traffic to a workload. | ||
| CIS Controls v8 | CIS-12 — Network Infrastructure Management | Supports runtime service availability checks and segmentation around broken workloads. |
| Recommendation — Instrument workload and dependency health checks across runtime paths. | ||
| OWASP ASVS | V16 — Security Logging and Error Handling | Error handling and observable failure states help distinguish app failure from container survival. |
| Recommendation — Log and surface application failure states that probes can detect. | ||
Practitioner Guidance
What to verify: Confirm that readiness failures are based on actual service reachability or request success, not just port-open checks or process existence. If the app can fail internally while the shell stays alive, the probe is too shallow.
What to measure: Track probe outcomes alongside request error rate, latency, and restart frequency. The useful signal is the correlation between “pod is Running” and “clients are failing,” because that is what reveals hidden partial outages.
Common mistake: Treating liveness as a substitute for readiness. Liveness should decide whether to restart; readiness should decide whether to receive traffic. If those roles blur, Kubernetes can preserve a broken workload longer than it should.
Practitioner takeaway: A healthy pod is not the same thing as a healthy service, so probe design must validate the workload’s real ability to serve traffic, not merely its survival.
Related resources from NHI Mgmt Group
- What are the signs that an audit logging pipeline is failing even when the application still looks healthy?
- What are the signs that a third-party connection is failing even though the integration still looks connected?
- What are the signs that a security data pipeline is failing even when logging appears healthy?
- What are the signs that SOC detection is failing even when dashboards look healthy?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 24, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
