Kubernetes Wazuh agent identity persistence in ephemeral workloads

At a glance

What this is: This is a deployment guide showing how Wazuh agents can preserve identity in Kubernetes so monitoring remains continuous across pod restarts.

Why it matters: It matters because IAM and security teams need telemetry continuity in ephemeral workloads, or agent re-creation becomes a visibility gap instead of a routine restart.

Context

Kubernetes makes workloads ephemeral, which means the monitoring layer has to cope with pod deletion, rescheduling, and replacement without losing identity. In this case, the primary governance gap is not access control but continuity of the agent identity that ties telemetry to the same monitored asset.

Wazuh’s article frames the problem around persistent registration, hostname stability, and volume-backed state for the agent running alongside containerized workloads. That is an identity lifecycle problem for non-human identities, because the agent must remain recognisable to the manager even when the pod instance changes.

The blog compares two deployment patterns, sidecar and included, both intended to keep the agent’s state intact across Kubernetes churn. The difference is operational trade-off, but the core requirement is the same: monitoring only works if the agent’s identity survives the workload lifecycle.

Key questions

Q: How should teams preserve monitoring continuity when Kubernetes pods are recreated?

A: Use persistent storage for the agent’s registration and configuration, and run it in a StatefulSet so rescheduling does not create a new monitoring identity. The goal is to keep the security platform tied to the same workload record even when the pod IP and runtime instance change.

Q: When does bundling a security agent into the application image create more risk than it reduces?

A: Bundling creates more risk when the organisation needs clear separation between the monitored workload and the monitoring component. It can simplify deployment, but it also blurs failure domains and makes agent lifecycle management depend on the application lifecycle.

Q: What breaks when an NHI agent loses its persisted state in Kubernetes?

A: The manager may treat the recreated pod as a new identity, which disrupts event lineage, history, and continuity of visibility. That turns routine pod churn into an operational blind spot because telemetry no longer maps cleanly to the same asset.

Q: How do sidecar and included deployment patterns differ for Kubernetes agents?

A: Sidecar deployment keeps the agent and application in separate containers with shared access to data, while included deployment bundles both into one image. Sidecar better preserves modularity, but included deployment can simplify rollout when teams accept tighter coupling.

Technical breakdown

Why persistent agent identity matters in Kubernetes

In Kubernetes, a pod is disposable, but a security agent often cannot be treated as disposable if its registration, hostname, and event history are used for monitoring continuity. The agent-manager relationship depends on stable identity mapping, not just a running process. When the pod is recreated, the manager needs enough persisted state to recognise the new instance as the same monitored endpoint. That usually means keeping configuration and registration artefacts outside the ephemeral container filesystem. Without that, every restart risks becoming a new identity event instead of a continuation of the same workload.

Practical implication: place agent state on persistent storage when telemetry continuity depends on the same identity surviving reschedules.

Sidecar deployment versus included deployment

A sidecar deployment keeps the monitoring agent and application in separate containers within the same pod, which preserves workload separation while allowing shared access to logs or runtime data through a mounted volume. An included deployment bundles the agent and application into one container image, which simplifies packaging but collapses separation boundaries. The first model is closer to container-native design because the agent remains distinct from the application. The second is easier to ship but makes the security and operational blast radius larger because the monitoring component and the monitored workload share the same runtime unit.

Practical implication: use sidecar patterns when you need separation of duties, and reserve included patterns for simpler environments that can tolerate tighter coupling.

How StatefulSets and persistent volumes preserve agent state

StatefulSets give Kubernetes a stable identity model for pods, including consistent naming and predictable volume attachment. Persistent volumes, such as those provisioned through Longhorn, hold the Wazuh agent’s registration keys and runtime data so the agent can recover with the same identity after restart or rescheduling. This matters because container filesystems are temporary, while identity records and monitoring continuity are not. The design pattern is therefore about anchoring NHI state outside the pod lifecycle so the manager sees continuity instead of re-enrollment churn.

Practical implication: pair StatefulSets with persistent volumes when the agent’s registration and configuration must survive pod replacement.

Breaches seen in the wild

• Secrets in Docker Hub images (RWTH Aachen study): A 2023 RWTH Aachen study found secrets in 8.5% of container images, and 275,269 internet hosts still using the leaked private keys.
• TeamTNT worm 2020: A TeamTNT cryptomining worm spread through exposed Docker APIs and stole AWS credential files from every host it infected, Cado Security found.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Persistent agent identity is the real control objective, not container uptime: Kubernetes can restart a pod in seconds, but that does not preserve the identity relationship a monitoring platform depends on. If registration keys, hostname, and configuration do not persist, the security control becomes blind to continuity. The practitioner conclusion is simple: treat the agent as a governed non-human identity, not as an interchangeable process.

Sidecar and included deployments solve different parts of the same governance problem: Sidecar keeps the agent separate from the application, which better preserves operational boundaries and monitoring clarity. Included deployment reduces deployment complexity, but it weakens workload separation and makes the monitoring component more tightly coupled to the application lifecycle. The implication is that deployment convenience should never override the identity and boundary model you need for monitoring.

Persistent volumes create identity continuity for ephemeral workloads: This is a textbook case of moving NHI state out of the pod filesystem and into infrastructure that survives rescheduling. Without that shift, every pod recreation risks being interpreted as a new agent rather than a continuation of the old one. Ephemeral workload identity debt: when identity state is tied to the pod rather than the workload record, monitoring continuity depends on luck. Practitioners should correct the state boundary, not just the deployment manifest.

Container separation still matters when the security agent is inside the workload: Bundling the agent into the same image as the application may be operationally convenient, but it reduces the clarity of failure domains and complicates lifecycle management. If the application and agent are managed as one unit, the organisation loses an opportunity to reason separately about telemetry health and application health. The practitioner conclusion is to align the deployment pattern with the level of governance separation the environment actually needs.

Identity lifecycle thinking now applies to runtime observability components: This article shows that agent registration, hostname stability, and recovery state are not implementation details. They are lifecycle properties that determine whether the monitoring plane can maintain an accurate record of the same workload over time. The practical takeaway is to manage observability agents with the same discipline used for other non-human identities.

From our research library:

• Red Hat’s Kubernetes Security Report found that nearly 9 in 10 organizations had at least 1 container or Kubernetes security incident in the last 12 months.

What this signals

Ephemeral workload identity is now a governance issue, not just a deployment detail: when the monitored process can disappear and reappear without losing its record, security teams need to manage the agent as a stateful identity object. That changes how observability, ownership, and lifecycle control are designed in Kubernetes.

Ephemeral workload identity debt: if the agent’s registration and hostname live only inside the pod, every restart risks becoming a new identity event. Practitioners should treat persistence of NHI state as part of the control plane for monitoring.

Kubernetes teams should expect greater pressure to align runtime observability with identity lifecycle governance. The practical question is no longer whether the agent runs, but whether it returns as the same managed identity after every reschedule.

For practitioners

• Design for persistent agent state Store Wazuh agent registration data, configuration, and hostname persistence on a volume that survives pod recreation so the manager keeps the same identity record.
• Use StatefulSets for identity continuity Run agent-backed workloads as StatefulSets when the monitoring relationship must survive rescheduling without forcing re-enrollment or record churn.
• Keep monitoring and application boundaries explicit Prefer sidecar deployment when you need the agent isolated from the application while still sharing logs or runtime data through a mounted volume.
• Validate reschedule recovery behavior Delete and recreate the pod in a test cluster, then confirm the agent returns with the same logical identity and uninterrupted event lineage.
• Separate simplicity from governance need Use the included deployment only when tighter bundling does not undermine separation, troubleshooting, or the identity lifecycle you need to govern.

Key takeaways

• Kubernetes pod churn can fragment monitoring unless the security agent keeps a persistent identity and state across restarts.
• StatefulSets and persistent volumes are the key mechanisms that let a recreated pod remain the same managed monitoring entity.
• The main design choice is not only deployment simplicity versus separation, but whether the observability layer can survive the workload lifecycle intact.

Key terms

• Workload Identity: The identity assigned to a software workload, such as a containerised application, serverless function, or microservice, enabling it to authenticate to other services without storing static credentials.
• StatefulSet: A StatefulSet is a Kubernetes controller that keeps stable network identities and ordered replacement behaviour for pods. For identity-sensitive monitoring, it helps preserve naming and attachment patterns that would otherwise change every time a pod is recreated.
• Sidecar Deployment: A sidecar deployment runs a helper container alongside the main application in the same pod. For security tools, it preserves separation of concerns while allowing the monitoring component to share local data and follow the application lifecycle closely.
• Included Deployment: Included deployment bundles the monitoring agent and the application into the same container image. It simplifies packaging, but it also couples their lifecycles and reduces the separation that often helps with troubleshooting, governance, and containment.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.