Rolling out a container runtime security or cloud workload protection platform (CWPP) means putting a privileged sensor on every node and, later, giving a tool the power to block deployments and kill processes in production, so a rushed rollout causes outages and an over-cautious one never leaves alert-only mode. This playbook sequences the rollout as a product class: scoping and success criteria, architecture and a pilot cluster, platform hardening, a detect-only baseline and tuning, SOC integration, and phased enforcement with rollback at every step. It is vendor-neutral and draws on Kubernetes documentation, NIST SP 800-190 and the NSA and CISA Kubernetes Hardening Guide. It is written for platform engineering leads, detection engineers and the tool owner. Runtime concepts and rule writing are in the Container Runtime Security Guide, and hardening the tool itself is in Securing the container runtime security and CWPP itself; this page links to both rather than repeating them.
Key takeaways
- Decide what the tool is for before you install it. Detection for the SOC, prevention at admission and in-kernel enforcement are different projects with different owners and risks.
- Pilot on a cluster you can break. Kernel versions, node images and the container runtime decide whether sensors work; find out on a non-production cluster that mirrors production.
- Secure the platform in the same sprint as the pilot. The sensor namespace, cluster roles and console access are set at install time and are hard to narrow later.
- Run detect-only long enough to know normal. Enforcing before you have a baseline of process, file and network behaviour produces outages and then exceptions that never close.
- Every enforcement step needs a rollback you have tested. Namespace by namespace, warn before enforce, Ignore before Fail, with a one-step pause.
Phase 0: scope, ownership and success criteria
- Name the use cases: runtime threat detection, drift detection (new executables not in the image), admission policy, in-kernel blocking, forensic capture, vulnerability context for running images. Most programmes start with the first two.
- Agree owners: the platform team owns the sensor and its namespace; detection engineering owns rules; the SOC owns triage; application teams own exceptions for their workloads. Write it down before the first alert fires.
- Inventory the estate: clusters, node operating systems and kernel versions, container runtimes, managed node pools versus serverless node options, and Windows nodes. Sensor support differs across all of these.
- Decide where it overlaps host EDR: if nodes already run an endpoint agent, agree which tool owns which signal. See Container Runtime Protection vs Host EDR.
- Set success criteria: coverage target, alert volume per cluster per day the SOC can absorb, mean time to triage, and the date by which the first namespaces move to enforcement.
Phase 1: architecture and pilot cluster
| Decision | Options | What to check in the pilot |
|---|---|---|
| Sensor technology | eBPF probe, kernel module, sidecar or runtime hook, managed agentless | Kernel versions supported; behaviour after a node image upgrade; CPU and memory overhead under load |
| Deployment method | Helm chart, operator, cloud add-on | Whether manifests can be rendered and reviewed before apply; how upgrades are delivered |
| Image source | Vendor registry or your mirror | Pull by digest from your mirror; signature verification works at admission |
| Telemetry path | Direct to vendor console, via an in-cluster relay, or to your own SIEM | Egress rules needed; what happens to events when the link is down |
| Admission component | None, validating only, validating and mutating | Latency added to pod creation; behaviour when the webhook is unavailable |
Pick a pilot cluster that mirrors production node images and runtime, but where a failed DaemonSet does not hurt customers. Use the DaemonSet RollingUpdate strategy, which is the Kubernetes default, with maxUnavailable: 1 (also the default) so a bad sensor version affects one node at a time. Record baseline node CPU, memory and pod start latency before install so you can measure the sensor's cost honestly.
Phase 2: secure the platform before it goes wide
Do this in the pilot, not after production. The detail is in Securing the container runtime security and CWPP itself; the rollout gate is that each item below is done and evidenced.
- Locked sensor namespace: a dedicated namespace labelled for the Privileged Pod Security level, with RBAC so only the platform team's deployment identity can create workloads in it. Every other namespace stays at Baseline or Restricted.
- Reviewed cluster roles: no secrets list or watch, nodes/proxy, bind, escalate, impersonate or namespace patch in the discovery role; response rights in a separate service account that is not yet bound.
- Console access: SSO with phishing-resistant MFA, scoped roles, one break-glass account, audit log export to your SIEM.
- Enrolment credentials: per-cluster, short-lived, stored in your secrets manager, never in a values file. See the Secrets Management Guide.
- Change control: sensor version, rules and exceptions deployed from a repository through a pipeline.
Phase 3: detect-only baseline and tuning
Run every rule in alert mode across the pilot, then across a first ring of production clusters. Two to four weeks per ring is a reasonable starting point (our recommendation); batch jobs and month-end processes need at least one full cycle.
- Start with high-fidelity rules: interactive shell in a container, new executable not present in the image, write below system binary directories, access to the service account token by an unexpected process, outbound connection to the API server from a workload that never makes one.
- Enrich every alert with cluster, namespace, pod, image digest, service account and owning team. An alert without an owner is not actionable.
- Tune with narrow exceptions: scope by image digest or workload, not by namespace or image name pattern, and give each exception an owner, reason and expiry.
- Build behavioural profiles for the workloads that will be enforced first, and review them with the application team.
- Track noise per rule: retire or rewrite rules whose true positive rate stays near zero after tuning. See the Detection as Code Guide.
Phase 4: SOC and response integration
| Integration | What good looks like | Test before go-live |
|---|---|---|
| SIEM | Alerts and sensor health events arrive with Kubernetes context and are correlated with API server audit logs | A test shell in a pilot pod reaches the SIEM within the agreed time |
| Case management | Alerts route to the owning team's queue with a runbook link | Ownership lookup works for every namespace |
| Identity response | Runbook covers revoking the workload's service account token and any cloud credentials it can reach | Tabletop with the identity team |
| Forensics | Capture of process tree, files and network data before a pod is deleted | Evidence handling agreed. See the Container and Kubernetes Forensics Guide |
| Coverage monitoring | Sensor heartbeat loss raises a security alert, not just a dashboard tile | Delete a sensor pod in the pilot and time the alert |
Containment for a confirmed compromise follows your cluster playbook; see the Kubernetes Cluster Compromise Playbook.
Phase 5: phased enforcement and rollback
Enforcement comes in three layers, each with its own ring sequence. Move one namespace or workload group at a time, starting with stateless services owned by a willing team.
| Layer | Step 1 | Step 2 | Step 3 | Rollback |
|---|---|---|---|---|
| Pod Security Admission | warn and audit at the target level | enforce: baseline with audit: restricted | enforce: restricted where workloads allow | Revert the namespace label through the pipeline |
| Tool admission webhook | Audit mode; failurePolicy: Ignore; one test namespace via namespaceSelector | Block mode for opted-in namespaces | Fail only where the webhook runs several replicas and the trade-off is approved | Switch to audit mode or narrow the selector; never delete the configuration by hand |
| Runtime response | Alert only | Kill or block a single high-fidelity behaviour, such as a shell in a production pod | Profile-based blocking for the workloads with reviewed profiles | Revert the policy version; pause enforcement with the kill switch |
- Exclude the cluster's essentials: follow the admission webhook good practices and do not match
kube-system,kube-node-leaseor the webhook's own namespace. - Sensor upgrades are changes too: roll new sensor versions ring by ring; if a version misbehaves, revert the DaemonSet to the previous revision through your pipeline and keep the history the controller retains for that.
- Bind the response role late: attach the response service account only to namespaces that have moved to enforcement.
- Test the kill switch in every ring before enforcement is switched on there.
Where identity fits
The rollout creates several non-human identities, and runtime alerts often end in a credential decision. Plan both from the start.
- Identities the tool adds: a discovery service account and a response service account per cluster, per-cluster enrolment credentials, console API keys for pipeline and SIEM integrations, and registry pull credentials for the mirrored sensor image. Register each with an owner, purpose and expiry in your NHI inventory before production. See the Service Account Security Guide.
- Identities the tool watches: many of the highest-value runtime detections are about workload credentials, such as a process reading the mounted service account token or calling the API server unexpectedly. Turn off token automounting for workloads that never call the API, so the signal gets cleaner as the rollout proceeds. See the Kubernetes Service Account Token Guide.
- Response includes revocation: killing a process does not invalidate a token it already copied. The SOC runbook must name who can revoke a workload's tokens and federated cloud credentials, and how fast. See the Kubernetes Workload Identity Checklist.
A typical rollout timeline
The durations below are NHI Mgmt Group's planning view for a mid-sized estate, not a standard; adjust them to your release cycles.
| Weeks | Work | Exit gate |
|---|---|---|
| 1 to 2 | Phase 0 scoping, inventory, owners | Signed scope and success criteria |
| 3 to 5 | Pilot install and platform hardening | Hardening gate evidenced; overhead measured |
| 6 to 10 | Detect-only in pilot and first production ring | Alert volume within the SOC's agreed capacity |
| 8 to 12 | SOC, identity and forensics integration | Each integration test passed |
| 12 onwards | Enforcement ring by ring | Rollback and kill switch tested per ring |
Decision guidance
- Small estate (a few clusters, one platform team): keep the scope to detection, drift and Pod Security Admission. Built-in admission (ValidatingAdmissionPolicy) may cover simple rules without a third-party webhook in the request path.
- Large or multi-tenant estate: invest in policy as code, per-team exception workflows and automated coverage reconciliation before enforcement; enforcement without them does not scale.
- Regulated or high-availability workloads: prefer
Ignoreplus a validating check and alert-heavy response, and enforce only behaviours with near-zero false positives. - Managed or serverless node options: where you cannot run a DaemonSet, confirm what the vendor supports for that node type before promising coverage.
- Already running host EDR on nodes: avoid two tools both killing processes; give enforcement to one and detection context to the other.
Detection and measurement
- Sensor coverage: nodes with a healthy sensor divided by nodes in the cluster, per ring (target: 100 percent of in-scope nodes).
- Alerts per cluster per day and true positive rate per rule, trended through tuning.
- Mean time from alert to triage decision in the SOC.
- Namespaces at each enforcement step (warn, enforce baseline, enforce restricted, runtime blocking) against the plan.
- Open exceptions, and exceptions past expiry (target: zero past expiry).
- Deployment failures or incidents caused by the tool, and time to roll back.
Framework mapping
| Rollout phase | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | ATT&CK threat addressed |
|---|---|---|---|
| Scope and inventory | CM-8 | ID.AM-02 | T1613 Container and Resource Discovery |
| Platform hardening | AC-6, AC-6(5), CM-6 | PR.AA-05, PR.PS-01 | T1610 Deploy Container |
| Detect-only baseline and tuning | SI-4, AU-6 | DE.CM-09 | T1609 Container Administration Command |
| SOC and response integration | IR-4, AU-6 | RS.MI-01 | T1552.007 Container API |
| Phased enforcement | CM-3, CM-7 | PR.PS-01 | T1611 Escape to Host |
| Coverage monitoring | SI-4 | PR.PS-04 | T1685 Disable or Modify Tools |
Common failure patterns
- Fleet-wide install on day one: an untested sensor version meets an unsupported kernel and crash-loops across every node.
- Enforcement before baseline: a blocking rule stops a nightly job, the business escalates, and the whole namespace gets a permanent exception.
- Webhook in the request path with
Failand one replica: a node drain takes the webhook down and no pod can start. - Alerts without owners: the SOC receives thousands of runtime alerts with no team mapping and starts ignoring them.
- Hardening deferred: the install chart's broad cluster roles and open sensor namespace stay in place because narrowing them later "might break something".
- Coverage assumed: new node pools join without sensors and nobody notices for months.
Practitioner checklist
- Write down use cases, owners and success criteria before installing anything.
- Inventory kernels, runtimes and node types, and confirm sensor support for each.
- Pilot on a production-like cluster with RollingUpdate and maxUnavailable 1, and record the overhead.
- Complete the platform hardening gate (namespace, roles, console, credentials, change control) in the pilot.
- Run detect-only per ring for at least one full business cycle and tune with narrow, expiring exceptions.
- Integrate with SIEM, case management, identity response and forensics, and test each.
- Enforce namespace by namespace: warn before enforce, Ignore before Fail, alert before block.
- Test rollback and the kill switch in every ring before enforcement there.
- Report coverage, noise, triage time and enforcement progress monthly.
Standards and references
- NIST: SP 800-190 Application Container Security Guide
- NSA and CISA: Kubernetes Hardening Guide
- Kubernetes: Pod Security Standards
- Kubernetes: Admission Webhook Good Practices
- Kubernetes: Perform a Rolling Update on a DaemonSet
- Kubernetes: Role Based Access Control Good Practices
- Kubernetes: Auditing
Related NHI Mgmt Group resources: Container Runtime Security Guide · Securing the container runtime security and CWPP itself · Container Runtime Protection vs Host EDR · Kubernetes cluster rollout playbook · CNAPP and CSPM platform rollout playbook