Join our Newsletter — 33% off our NHI Course
Home› Guides› Kubernetes cluster rollout playbook: phased implementation with security…
Guide Cloud Security

Kubernetes cluster rollout playbook: phased implementation with security built in

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 15 min read
On this page

Most of the weaknesses a Kubernetes cluster carries for years are decided in the first few weeks, when the first admin credential is issued, the first namespace is created and the first team is told to "just deploy". This playbook sequences the rollout of a new cluster or cluster fleet, self-managed or delivered as a managed service, so that baseline configuration, identity and RBAC, admission policy and observability are in place before application teams arrive. It is vendor-neutral and written for platform engineering, cloud security and SRE teams planning a first production platform or a consolidated second one. It is about order, gates and evidence: control depth lives in the Kubernetes Security Guide and the specialist guides linked from each phase, and this page does not repeat them.

Key takeaways

  • Bootstrap credentials are break-glass, not admin accounts. The Kubernetes security checklist says system:masters should only be used as a break-glass mechanism after bootstrapping, so vault it before anyone logs in day to day.
  • Admission policy goes in before workloads. Turn on Pod Security Admission in warn and audit mode at platform build, so the move to enforce is a measured step rather than an outage.
  • Audit logging is off until you configure it. Without --audit-policy-file the API server logs no audit events, so on self-managed control planes this is a build task, not a later improvement.
  • Every workload gets its own identity. Dedicated service accounts, no token automount where the API is not called, and federation to cloud services instead of stored keys.
  • Handover is a gate, not a date. Operations accept the cluster only when the health check metrics, runbooks and an exercised compromise playbook exist.

The rollout at a glance

Each phase ends in a gate with named evidence. Durations are indicative planning ranges for a single production cluster and its non-production twin; a fleet adds time to Phase 1 and Phase 5, not to the others.

PhaseGoalGate evidenceIndicative duration
0. PlanScope, ownership, tenancy model, version and support policySigned architecture decision record, named platform owner, workload inventory2 to 4 weeks
1. BaselineHardened control plane, nodes, etcd and networkCIS Kubernetes Benchmark scan with exceptions recorded, encryption at rest verified2 to 4 weeks
2. Identity and RBACHuman access via SSO, workload identities, break-glass vaultedRBAC review showing no standing cluster-admin for people, break-glass test record2 to 3 weeks
3. AdmissionPod Security Admission and policy engine, audit then enforceViolation report trending to zero, enforce labels on all application namespaces3 to 6 weeks, overlapping Phase 4
4. ObservabilityAudit logs, runtime signals and alerts flowing to the SOCDetection test results, retention confirmed, log access restricted2 to 4 weeks
5. Pilot and handoverFirst tenant live, runbooks accepted by operationsPilot sign-off, tabletop of the compromise playbook, health check baseline3 to 6 weeks

Phase 0: plan

  1. Tenancy model: decide whether teams share clusters with namespace isolation or get dedicated clusters. Namespaces are a policy boundary, not a security boundary against a hostile tenant; workloads of different trust levels (internet-facing, payment, untrusted build jobs) should be separated by node pools or clusters, using taints, tolerations and node affinity where they share a cluster.
  2. Control plane ownership: with a managed control plane the provider runs etcd and the API server, but the customer still owns RBAC, admission, workload configuration, node images where self-managed and the cloud identities bound to the cluster. Write this split down.
  3. Version and patch policy: the Kubernetes project maintains release branches for the three most recent minor releases, and releases from 1.19 onwards receive roughly one year of patch support. Set an internal rule such as "no cluster more than one minor version behind the newest supported release" and budget an upgrade at least every year.
  4. Workload inventory: list the applications that will land in the first two waves, what they need from the API (most need nothing), which cloud services they call and which secrets they hold today. Record metadata only, never values.
  5. Policy-as-code repository: create the repository that will hold cluster configuration, RBAC, admission policies and audit policy, with branch protection and review. Everything after this phase is a change to that repository.

The inventory supports NIST CSF 2.0 ID.AM-02 and NIST SP 800-53 Rev. 5 CM-8 System Component Inventory. If you are writing the policy that governs this platform at the same time, start from the Container and Kubernetes Security Policy Template.

Phase 1: baseline the platform

Build the control plane and node configuration from code, then measure it against the CIS Kubernetes Benchmark. CIS publishes a generic Kubernetes benchmark and separate benchmarks for the major managed distributions; use the one that matches your platform, because many control plane checks do not apply where the provider runs the control plane.

Baseline itemSetting or checkSource
ExposureKubernetes API, kubelet API and etcd are not exposed publicly on the internetKubernetes security checklist
etcdAccess limited, mutual TLS, a certificate authority unique to etcdKubernetes security checklist
Root CAOffline CA, or a managed online CA with effective access controlsKubernetes security checklist
Secrets at restEncryptionConfiguration passed with --encryption-provider-config; prefer the kms v2 provider, stable since 1.29Encrypting confidential data at rest
Admission pluginsKeep the defaults on (including PodSecurity and CertificateSubjectRestriction); add NodeRestriction and DenyServiceExternalIPsKubernetes security checklist
Controller credentialskube-controller-manager runs with --use-service-account-credentialsKubernetes security checklist
NetworkDefault-deny network policy template applied to every new namespaceKubernetes Network Policy Guide

Two encryption details catch teams out. The default provider is identity, which stores Secret data in etcd as plain text, and turning on encryption only affects objects written afterwards: existing Secrets must be rewritten, and the identity fallback removed once they are. Keys held in the configuration file on the control plane host give only modest protection; a KMS provider keeps the key encryption key outside the cluster. Verify the result with the steps in How to check Kubernetes secrets encryption at rest.

Gate: benchmark scan attached to the change record, each failed check either fixed or carried as a dated exception with an owner. Rollback in this phase is cheap: rebuild from code.

Phase 2: identity and RBAC

People

  • Single sign-on: authenticate administrators and developers through the corporate identity provider with phishing-resistant MFA, and bind RBAC to groups rather than individual users.
  • Break-glass: the bootstrap kubeconfig that carries system:masters bypasses every RBAC check and cannot be revoked by deleting bindings. Store it in a vault with dual control, test it once, and alert on any use. Follow the Break-Glass Account Guide.
  • Day-to-day admin: a dedicated group bound to elevated roles through just-in-time elevation, not standing membership. Namespaced Roles and RoleBindings by default; ClusterRoleBindings by exception.

Permissions to keep out of everyday roles

The Kubernetes RBAC good practices page singles out permissions that lead to escalation. Check every role in the policy repository for them before Phase 3.

PermissionWhy it mattersRollout rule
escalate, bind on rolesAllow creating roles or bindings with rights the user does not holdPlatform team only, through the policy repository
impersonateGains the rights of other users, groups or service accountsBreak-glass and named tooling only
nodes/proxyCommand execution in every pod on a node, bypassing audit logging and admission control; even get is not read-onlyMonitoring agents with a documented need
list or watch on SecretsEffectively reveal Secret contentsNever in developer roles; grant get on named Secrets
create on pods or workloadsCan mount any Secret or run as any service account in the namespaceScope by namespace and constrain with admission policy
Webhook configuration objectsCan intercept or block API requests cluster-widePlatform team only

Workloads

Each application gets a dedicated service account, automountServiceAccountToken: false where it does not call the API, and bound, expiring projected tokens where it does, as the security checklist requires. Cloud access goes through workload identity federation rather than stored access keys. The Kubernetes Workload Identity Checklist covers the settings; the Kubernetes NHI Security Guide covers service account governance across the cluster's life.

Gate: output of the review in How to audit Kubernetes cluster admin bindings showing no person with standing cluster-admin, and a dated break-glass test.

Phase 3: admission policy, audit before enforce

Pod Security Admission applies one of three levels (privileged, baseline, restricted) per namespace in three modes. enforce rejects violating pods, audit adds an annotation to the audit event, warn returns a warning to the client. The labels take the form pod-security.kubernetes.io/<MODE>: <LEVEL>, with an optional <MODE>-version label to pin the policy to a Kubernetes minor version.

  1. Week 1: label all application namespaces warn and audit at restricted, enforce at baseline. Pin the version so a cluster upgrade does not change policy unannounced.
  2. Weeks 1 to 3: report audit annotations per namespace; fix manifests in the pilot applications; record any genuine exception.
  3. Weeks 3 to 6: move enforce to restricted namespace by namespace once its violation count reaches zero for a full release cycle.
  4. Policy engine: add a validating admission policy or webhook engine for what Pod Security Admission does not cover: approved registries, image digests or signatures, required labels, no hostPath outside system namespaces. Run it in audit mode first, exactly as above.
  5. Rollback: revert the namespace label or policy in the repository. Because exemptions are applied by username, runtime class or namespace, keep them in code with an owner and expiry. The project warns against exempting controller service accounts, since that implicitly exempts anyone who can create the workload resource.

Admission webhooks extend the API server, so secure them as tier 0: their own namespace, restricted RBAC and a deliberate failure policy. Design depth is in the Kubernetes Admission Control Guide; teams coming from the removed PodSecurityPolicy should read PodSecurityPolicy to Pod Security Admission.

Phase 4: observability

On a self-managed control plane, pass an audit policy with --audit-policy-file and a backend: --audit-log-path for files, or --audit-webhook-config-file to send events to a remote collector. Audit levels are None, Metadata, Request and RequestResponse, and the first matching rule wins. A workable starting policy logs Secrets, ConfigMaps and token requests at Metadata only (so values never reach the log), RBAC and webhook configuration changes at RequestResponse, and everything else at Metadata. On a managed platform, enable the provider's control plane audit log export instead.

SignalWhy it is in the first alert setATT&CK
Use of the break-glass identity or any system:masters requestShould never happen outside a declared incidentT1078.004 Cloud Accounts
pods/exec and pods/attach in production namespacesInteractive access to running workloadsT1609 Container Administration Command
Privileged pod or hostPath creation attemptsPrecursor to node takeoverT1610 Deploy Container, T1611 Escape to Host
Secret list across namespaces, unusual token requestsCredential harvesting from the APIT1552.007 Container API, T1528 Steal Application Access Token
New ClusterRoleBinding or webhook configurationPersistence and privilege escalationT1078.004 Cloud Accounts
Anonymous or unexpected identities enumerating resourcesDiscovery before attackT1613 Container and Resource Discovery

Audit logs themselves must be protected from general access, as the checklist says, and shipped off the cluster to a store the platform team cannot alter. Query patterns are in the Kubernetes Audit Log Detection Guide.

Gate: each alert above fired once in non-production by a controlled test, received by the SOC and closed with a runbook.

Phase 5: pilot and handover

  1. Pilot tenant: one team with a real but non-critical service, onboarded through the standard namespace template (quotas, network policy, Pod Security labels, service account, RBAC group bindings).
  2. Backup and restore: prove restore of cluster state and persistent volumes before the first critical workload. See Kubernetes Backup and Recovery.
  3. Incident readiness: run a tabletop of the Kubernetes Cluster Compromise Playbook with platform, SOC and the pilot team, including credential rotation for the cluster CA and service account signing keys.
  4. Health check baseline: take the first reading of the Kubernetes Platform Health Check and agree its cadence with operations.
  5. Handover: operations sign acceptance of runbooks for upgrades, certificate rotation, encryption key rotation, node patching and break-glass use. Later waves follow only after the pilot has run one minor upgrade or a full patch cycle cleanly.

Where identity fits

A cluster is an identity system in its own right. The API server trusts certificates, bearer tokens, OIDC tokens and service account tokens; controllers, operators, CI pipelines, GitOps agents and monitoring tools all hold credentials that often outlive the people who created them. Treat the CI or GitOps deployer as the most privileged non-human identity on the platform: scope it per namespace, federate it rather than storing a kubeconfig, and review it with the same cadence as human administrators. The OWASP Kubernetes Top 10 (2025) lists K02 Overly Permissive Authorization Configurations, K03 Secrets Management Failures, K08 Cluster To Cloud Lateral Movement and K09 Broken Authentication Mechanisms, four of its ten risks, as identity problems. For stale identities after go-live, use How to find unused Kubernetes service accounts.

Small and large organisations

DecisionSmall team, one or two clustersLarge organisation, fleet
Control planeManaged control plane, so etcd and API server patching sit with the providerManaged by default; self-managed only where residency or air gaps require it, with a dedicated control plane team
TenancyCluster per environment, namespaces per applicationCluster per trust tier or business unit, namespaces per team
Policy enginePod Security Admission plus built-in validating admission policiesCentral policy engine with policies distributed from one repository to all clusters
LoggingProvider audit export to the existing SIEMCentral pipeline with per-cluster tagging and fleet-wide detections
UpgradesAnnual, planned with the provider calendarRolling waves, canary clusters first, version skew tracked as a metric

Detection and measurement

  • Standing cluster-admin holders: people with permanent cluster-wide admin. Target zero, excluding the vaulted break-glass identity.
  • Enforce coverage: percentage of application namespaces labelled enforce: restricted. Target 100 per cent with dated exceptions.
  • Secret encryption: Secrets confirmed encrypted in etcd or by the provider. Target 100 per cent.
  • Token automount rate: pods with a service account token mounted that never call the API. Trend downward each wave.
  • Version currency: clusters on a minor release the project still maintains. Target 100 per cent.
  • Alert test pass rate: first-set detections that fired and reached the SOC in the last quarterly test.

Common failure patterns

  • Admin kubeconfig shared on a wiki: the bootstrap credential becomes the team credential, and because it carries system:masters no RBAC change can contain it.
  • Enforce on day one: moving straight to restricted breaks the pilot, the label is removed in a hurry and never comes back. Warn and audit first.
  • Encryption switched on but never applied: the configuration is in place, the existing Secrets were never rewritten, and the identity fallback is still there.
  • Monitoring agent with nodes/proxy: a broad read role turns out to allow command execution that bypasses audit and admission.
  • Audit log present, nobody reads it: events are written to the control plane host and rotated away; no alert set was ever tested.
  • Exempt the controller: a Pod Security exemption for a controller service account quietly exempts every user who can create a Deployment.

Framework mapping

PhaseNIST SP 800-53 Rev. 5NIST CSF 2.0CIS Controls v8.1ISO/IEC 27001:2022 Annex A
0. PlanCM-8ID.AM-021, 2A.5.9
1. BaselineCM-2, CM-6, CM-7, SC-28, SC-12, SI-2PR.PS-01, PR.PS-024, 3, 7A.8.9, A.8.24, A.8.8
2. Identity and RBACAC-2, AC-6, AC-6(5), IA-5PR.AA-01, PR.AA-055, 6A.5.15, A.5.18, A.8.2, A.8.5
3. AdmissionCM-7, CM-6PR.PS-014, 16A.8.9
4. ObservabilityAU-2, AU-6, AU-9, AC-6(9)PR.PS-04, DE.CM-098, 13A.8.15, A.8.16
5. HandoverCA-7DE.CM-0911, 17A.5.24, A.8.13

Practitioner checklist

  • Tenancy model, control plane ownership split and version policy recorded in an architecture decision record.
  • All cluster configuration, RBAC, admission and audit policy held in a reviewed repository.
  • CIS benchmark scan for the matching distribution attached, exceptions dated and owned.
  • Secrets encrypted at rest with a KMS provider, existing Secrets rewritten, identity fallback removed.
  • system:masters credential vaulted, tested and alerted on; no standing human cluster-admin.
  • Escalation permissions (escalate, bind, impersonate, nodes/proxy, Secret list) absent from everyday roles.
  • Every workload on a dedicated service account, automount off where unused, cloud access federated.
  • Pod Security Admission in warn and audit from build; enforce restricted per namespace when clean.
  • Audit policy live, logs off-cluster and access-restricted, first alert set tested.
  • Restore tested, compromise playbook exercised, health check baseline taken before handover.

Standards and references

Related NHI Mgmt Group resources: Kubernetes Security Guide, Kubernetes Admission Control Guide, Kubernetes Cluster Compromise Playbook, Kubernetes Workload Identity Checklist, Kubernetes Platform Health Check.

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.