Kubernetes Wazuh agent identity persistence in ephemeral workloads

At a glance

What this is: This is a Wazuh guide to keeping agent identity stable in Kubernetes by using sidecar and included deployment patterns with persistent storage.

Why it matters: It matters because IAM and security teams need monitoring agents, service accounts, and workload identities to remain recognisable across pod recreation without losing visibility or audit continuity.

👉 Read Wazuh's guide to persistent Wazuh agent deployments in Kubernetes

Context

Kubernetes makes workload identity temporary by design, but security monitoring still needs a stable identity to preserve event history, ownership, and continuity across restarts. In this article, the primary governance problem is how a non-human identity can remain recognisable when the underlying pod is recreated, rescheduled, or replaced.

That matters for the Wazuh agent because if the agent re-registers as a new identity every time the pod changes, the monitoring trail fragments. The post focuses on using StatefulSets and persistent volumes so the agent keeps its registration key and hostname, which is a practical workload identity pattern rather than a generic container tip.

This is a typical problem in Kubernetes environments that rely on ephemeral workloads and shared clusters, especially when teams want the monitoring layer to survive ordinary pod churn without losing correlation.

Key questions

Q: How should security teams preserve workload identity in Kubernetes monitoring setups?

A: Use persistent storage for registration state, stable naming through StatefulSets, and a deployment model that keeps the agent recognisable after pod recreation. The goal is not to make pods permanent, but to make the monitored workload continuously attributable across routine cluster churn.

Q: What breaks when a monitoring agent loses its identity after pod rescheduling?

A: Event correlation fragments, host records reset, and security teams lose continuity across restarts even though the application has not materially changed. That creates blind spots in detection and weakens audit evidence because the same workload is now treated as a new subject.

Q: How do sidecar and included deployment patterns differ for Kubernetes agents?

A: Sidecar deployment keeps the agent and application in separate containers with shared access to data, while included deployment bundles both into one image. Sidecar better preserves modularity, but included deployment can simplify rollout when teams accept tighter coupling.

Q: Why do NHI controls matter for container monitoring agents?

A: Because the agent is a non-human identity whose credentials, registration state, and lifecycle determine whether the monitoring record remains trustworthy. Without NHI governance, ephemeral containers can turn one workload into many apparent identities, which weakens visibility and response.

Technical breakdown

Why Kubernetes pod churn breaks workload identity continuity

Kubernetes treats pods as replaceable objects, which is useful for scaling but awkward for identity-bound tooling. If a monitoring agent stores its registration state inside the pod filesystem, pod deletion or rescheduling can force re-enrolment, new host records, and broken event correlation. The technical issue is not logging alone, but identity continuity across ephemeral execution units. StatefulSets help by preserving ordinal naming, while persistent volumes preserve registration data that would otherwise vanish with the container.

Practical implication: keep identity-bearing agent state outside the ephemeral pod filesystem if you need stable monitoring continuity.

How sidecar deployment preserves agent and application separation

A sidecar pattern places the monitoring agent and the application in separate containers within the same pod, usually with a shared volume for log access. That preserves operational separation while still allowing the agent to observe local runtime data. In identity terms, the agent keeps its own lifecycle while sharing scheduling fate with the application. The approach is cleaner than baking the agent into the application image because it avoids coupling monitoring logic to app release cycles.

Practical implication: use sidecars when you want shared visibility without collapsing the application and the monitoring agent into one lifecycle.

What included deployment changes about identity and recovery

An included deployment bundles the Wazuh agent and application into one container image, which simplifies rollout but increases coupling. The container now carries both application and monitoring state, so recovery is less modular and troubleshooting becomes harder when one component fails. This pattern can still preserve agent identity if persistent storage is used, but it weakens the separation that container security teams usually want. The main architectural trade-off is simplicity versus containment.

Practical implication: only bundle the agent when deployment simplicity matters more than clean separation and independent operational recovery.

NHI Mgmt Group analysis

Persistent workload identity is the real control objective here, not pod permanence. Kubernetes pods are intentionally disposable, so identity must survive outside the pod lifecycle if monitoring is to remain trustworthy. The article shows that persistence belongs to the agent's registration state, not to the pod object itself. For practitioners, the key question is whether the monitoring identity can outlive ordinary cluster churn without being reissued as a new entity.

StatefulSet plus persistent volume is a workload identity pattern, not just a storage pattern. The combination gives the agent a stable name and retained registration data after restart or reschedule. That is the minimum structure needed for continuous event correlation in dynamic clusters. Without that persistence layer, monitoring becomes a sequence of unrelated agent records rather than one governed identity.

Ephemeral runtime, persistent identity: this is the named concept the article makes visible. The workload can be short-lived while the identity must remain durable enough for the manager to recognise continuity, which is exactly the opposite of how many teams still model containers. That matters for NHI governance because access, trust, and telemetry all depend on continuity of the subject being observed.

Bundling the agent with the application simplifies deployment but weakens identity governance boundaries. Included deployment collapses two operational lifecycles into one image, which can obscure who owns monitoring state and when it changes. In governance terms, this is where lifecycle clarity matters as much as technical functionality. Teams should treat agent placement as an identity design decision, not only a deployment convenience.

This pattern is a reminder that workload observability is an identity problem before it is a tooling problem. If the monitoring agent cannot be re-associated with the same workload after rescheduling, the security record loses continuity even when the application is unchanged. That undermines incident review, compliance evidence, and detection fidelity. Practitioners should therefore evaluate Kubernetes monitoring through workload identity continuity first, and implementation style second.

From our research:

• 88.5% of organisations acknowledge that their non-human IAM practices lag behind or are merely on par with their human identity and access management efforts, according to the 2024 Non-Human Identity Security Report.
• Only 19.6% of security professionals express strong confidence in their organisation's ability to securely manage non-human workload identities, which helps explain why Kubernetes monitoring identities remain fragile.
• For a broader baseline on machine identity governance, see the NHI Lifecycle Management Guide for provisioning, rotation, and offboarding patterns that apply beyond Kubernetes.

What this signals

Ephemeral runtime, persistent identity: Kubernetes monitoring will keep exposing the gap between infrastructure scheduling and identity governance until teams treat registration state as a controlled lifecycle asset. That shift matters because a recreated pod is operationally new but analytically should still be the same monitored subject, and the control plane has to preserve that continuity.

With 35.6% of organisations already naming consistent access across hybrid and multi-cloud environments as their top NHI challenge, persistent identity for workload agents becomes part of a wider governance problem rather than a niche deployment choice. Teams that standardise persistent volumes, stable naming, and agent state handling will be better positioned to keep security telemetry coherent across dynamic infrastructure.

For practitioners building this pattern into their programme, the next step is to align monitoring design with the Ultimate Guide to NHIs and the SPIFFE workload identity specification where workload identity attestation and continuity are relevant.

For practitioners

• Preserve agent registration outside the pod lifecycle Store registration keys, hostname state, and other identity-bearing files on persistent volumes so a recreated pod is still recognised as the same monitored workload.
• Prefer StatefulSets for identity-sensitive monitoring agents Use StatefulSets when the security tool needs stable naming and predictable reattachment after rescheduling, especially in ephemeral Kubernetes clusters.
• Separate agent lifecycle from application release cycles Deploy the agent as a sidecar when you need independent monitoring continuity without rebuilding the application image for every security change.
• Test pod recreation as an identity continuity check Delete and recreate the pod, then verify the manager still correlates the workload to the same agent record instead of creating a new identity.
• Avoid collapsing monitoring and application state into one image unless necessary Bundled containers reduce deployment steps, but they also make recovery, ownership, and troubleshooting harder when the monitoring layer needs separate governance.

Key takeaways

• Kubernetes pod churn can break monitoring continuity unless the agent's identity state survives outside the pod.
• The article's main lesson is that workload observability depends on persistent identity, not just persistent storage.
• Teams should choose deployment patterns based on whether they need modular separation, stable agent identity, or both.

Key terms

• Workload Identity: The identity assigned to a software workload — such as a containerised application, serverless function, or microservice — enabling it to authenticate to other services without storing static credentials.
• StatefulSet: A StatefulSet is a Kubernetes controller that keeps stable network identities and ordered replacement behaviour for pods. For identity-sensitive monitoring, it helps preserve naming and attachment patterns that would otherwise change every time a pod is recreated.
• Persistent Volume: A persistent volume is storage that remains available after a pod is deleted or rescheduled. For NHI use cases, it often holds registration keys, configuration, and state that define whether the same workload is still being observed as the same identity.
• Sidecar Deployment: A sidecar deployment runs a helper container alongside the main application in the same pod. For security tools, it preserves separation of concerns while allowing the monitoring component to share local data and follow the application lifecycle closely.

What's in the full article

Wazuh's full blog post covers the operational deployment details this analysis intentionally leaves for the source:

• The exact Kubernetes manifests for sidecar and included deployment patterns, including StatefulSet and persistent volume configuration.
• The step-by-step Longhorn and K3s setup used to make the lab environment work in practice.
• The full init-container logic for preserving agent files, permissions, and registration state across pod recreation.
• The complete pod recreation test results showing how the Wazuh manager re-associates the same agent identity after rescheduling.

👉 Wazuh's full post shows the manifest details and resilience tests for both deployment patterns.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.