Most of the weaknesses a Kubernetes cluster carries for years are decided in the first few weeks, when the first admin credential is issued, the first namespace is created and the first team is told to "just deploy". This playbook sequences the rollout of a new cluster or cluster fleet, self-managed or delivered as a managed service, so that baseline configuration, identity and RBAC, admission policy and observability are in place before application teams arrive. It is vendor-neutral and written for platform engineering, cloud security and SRE teams planning a first production platform or a consolidated second one. It is about order, gates and evidence: control depth lives in the Kubernetes Security Guide and the specialist guides linked from each phase, and this page does not repeat them.
Key takeaways
- Bootstrap credentials are break-glass, not admin accounts. The Kubernetes security checklist says
system:mastersshould only be used as a break-glass mechanism after bootstrapping, so vault it before anyone logs in day to day. - Admission policy goes in before workloads. Turn on Pod Security Admission in warn and audit mode at platform build, so the move to enforce is a measured step rather than an outage.
- Audit logging is off until you configure it. Without
--audit-policy-filethe API server logs no audit events, so on self-managed control planes this is a build task, not a later improvement. - Every workload gets its own identity. Dedicated service accounts, no token automount where the API is not called, and federation to cloud services instead of stored keys.
- Handover is a gate, not a date. Operations accept the cluster only when the health check metrics, runbooks and an exercised compromise playbook exist.
The rollout at a glance
Each phase ends in a gate with named evidence. Durations are indicative planning ranges for a single production cluster and its non-production twin; a fleet adds time to Phase 1 and Phase 5, not to the others.
| Phase | Goal | Gate evidence | Indicative duration |
|---|---|---|---|
| 0. Plan | Scope, ownership, tenancy model, version and support policy | Signed architecture decision record, named platform owner, workload inventory | 2 to 4 weeks |
| 1. Baseline | Hardened control plane, nodes, etcd and network | CIS Kubernetes Benchmark scan with exceptions recorded, encryption at rest verified | 2 to 4 weeks |
| 2. Identity and RBAC | Human access via SSO, workload identities, break-glass vaulted | RBAC review showing no standing cluster-admin for people, break-glass test record | 2 to 3 weeks |
| 3. Admission | Pod Security Admission and policy engine, audit then enforce | Violation report trending to zero, enforce labels on all application namespaces | 3 to 6 weeks, overlapping Phase 4 |
| 4. Observability | Audit logs, runtime signals and alerts flowing to the SOC | Detection test results, retention confirmed, log access restricted | 2 to 4 weeks |
| 5. Pilot and handover | First tenant live, runbooks accepted by operations | Pilot sign-off, tabletop of the compromise playbook, health check baseline | 3 to 6 weeks |
Phase 0: plan
- Tenancy model: decide whether teams share clusters with namespace isolation or get dedicated clusters. Namespaces are a policy boundary, not a security boundary against a hostile tenant; workloads of different trust levels (internet-facing, payment, untrusted build jobs) should be separated by node pools or clusters, using taints, tolerations and node affinity where they share a cluster.
- Control plane ownership: with a managed control plane the provider runs etcd and the API server, but the customer still owns RBAC, admission, workload configuration, node images where self-managed and the cloud identities bound to the cluster. Write this split down.
- Version and patch policy: the Kubernetes project maintains release branches for the three most recent minor releases, and releases from 1.19 onwards receive roughly one year of patch support. Set an internal rule such as "no cluster more than one minor version behind the newest supported release" and budget an upgrade at least every year.
- Workload inventory: list the applications that will land in the first two waves, what they need from the API (most need nothing), which cloud services they call and which secrets they hold today. Record metadata only, never values.
- Policy-as-code repository: create the repository that will hold cluster configuration, RBAC, admission policies and audit policy, with branch protection and review. Everything after this phase is a change to that repository.
The inventory supports NIST CSF 2.0 ID.AM-02 and NIST SP 800-53 Rev. 5 CM-8 System Component Inventory. If you are writing the policy that governs this platform at the same time, start from the Container and Kubernetes Security Policy Template.
Phase 1: baseline the platform
Build the control plane and node configuration from code, then measure it against the CIS Kubernetes Benchmark. CIS publishes a generic Kubernetes benchmark and separate benchmarks for the major managed distributions; use the one that matches your platform, because many control plane checks do not apply where the provider runs the control plane.
| Baseline item | Setting or check | Source |
|---|---|---|
| Exposure | Kubernetes API, kubelet API and etcd are not exposed publicly on the internet | Kubernetes security checklist |
| etcd | Access limited, mutual TLS, a certificate authority unique to etcd | Kubernetes security checklist |
| Root CA | Offline CA, or a managed online CA with effective access controls | Kubernetes security checklist |
| Secrets at rest | EncryptionConfiguration passed with --encryption-provider-config; prefer the kms v2 provider, stable since 1.29 | Encrypting confidential data at rest |
| Admission plugins | Keep the defaults on (including PodSecurity and CertificateSubjectRestriction); add NodeRestriction and DenyServiceExternalIPs | Kubernetes security checklist |
| Controller credentials | kube-controller-manager runs with --use-service-account-credentials | Kubernetes security checklist |
| Network | Default-deny network policy template applied to every new namespace | Kubernetes Network Policy Guide |
Two encryption details catch teams out. The default provider is identity, which stores Secret data in etcd as plain text, and turning on encryption only affects objects written afterwards: existing Secrets must be rewritten, and the identity fallback removed once they are. Keys held in the configuration file on the control plane host give only modest protection; a KMS provider keeps the key encryption key outside the cluster. Verify the result with the steps in How to check Kubernetes secrets encryption at rest.
Gate: benchmark scan attached to the change record, each failed check either fixed or carried as a dated exception with an owner. Rollback in this phase is cheap: rebuild from code.
Phase 2: identity and RBAC
People
- Single sign-on: authenticate administrators and developers through the corporate identity provider with phishing-resistant MFA, and bind RBAC to groups rather than individual users.
- Break-glass: the bootstrap kubeconfig that carries
system:mastersbypasses every RBAC check and cannot be revoked by deleting bindings. Store it in a vault with dual control, test it once, and alert on any use. Follow the Break-Glass Account Guide. - Day-to-day admin: a dedicated group bound to elevated roles through just-in-time elevation, not standing membership. Namespaced Roles and RoleBindings by default; ClusterRoleBindings by exception.
Permissions to keep out of everyday roles
The Kubernetes RBAC good practices page singles out permissions that lead to escalation. Check every role in the policy repository for them before Phase 3.
| Permission | Why it matters | Rollout rule |
|---|---|---|
escalate, bind on roles | Allow creating roles or bindings with rights the user does not hold | Platform team only, through the policy repository |
impersonate | Gains the rights of other users, groups or service accounts | Break-glass and named tooling only |
nodes/proxy | Command execution in every pod on a node, bypassing audit logging and admission control; even get is not read-only | Monitoring agents with a documented need |
list or watch on Secrets | Effectively reveal Secret contents | Never in developer roles; grant get on named Secrets |
create on pods or workloads | Can mount any Secret or run as any service account in the namespace | Scope by namespace and constrain with admission policy |
| Webhook configuration objects | Can intercept or block API requests cluster-wide | Platform team only |
Workloads
Each application gets a dedicated service account, automountServiceAccountToken: false where it does not call the API, and bound, expiring projected tokens where it does, as the security checklist requires. Cloud access goes through workload identity federation rather than stored access keys. The Kubernetes Workload Identity Checklist covers the settings; the Kubernetes NHI Security Guide covers service account governance across the cluster's life.
Gate: output of the review in How to audit Kubernetes cluster admin bindings showing no person with standing cluster-admin, and a dated break-glass test.
Phase 3: admission policy, audit before enforce
Pod Security Admission applies one of three levels (privileged, baseline, restricted) per namespace in three modes. enforce rejects violating pods, audit adds an annotation to the audit event, warn returns a warning to the client. The labels take the form pod-security.kubernetes.io/<MODE>: <LEVEL>, with an optional <MODE>-version label to pin the policy to a Kubernetes minor version.
- Week 1: label all application namespaces
warnandauditatrestricted,enforceatbaseline. Pin the version so a cluster upgrade does not change policy unannounced. - Weeks 1 to 3: report audit annotations per namespace; fix manifests in the pilot applications; record any genuine exception.
- Weeks 3 to 6: move
enforcetorestrictednamespace by namespace once its violation count reaches zero for a full release cycle. - Policy engine: add a validating admission policy or webhook engine for what Pod Security Admission does not cover: approved registries, image digests or signatures, required labels, no
hostPathoutside system namespaces. Run it in audit mode first, exactly as above. - Rollback: revert the namespace label or policy in the repository. Because exemptions are applied by username, runtime class or namespace, keep them in code with an owner and expiry. The project warns against exempting controller service accounts, since that implicitly exempts anyone who can create the workload resource.
Admission webhooks extend the API server, so secure them as tier 0: their own namespace, restricted RBAC and a deliberate failure policy. Design depth is in the Kubernetes Admission Control Guide; teams coming from the removed PodSecurityPolicy should read PodSecurityPolicy to Pod Security Admission.
Phase 4: observability
On a self-managed control plane, pass an audit policy with --audit-policy-file and a backend: --audit-log-path for files, or --audit-webhook-config-file to send events to a remote collector. Audit levels are None, Metadata, Request and RequestResponse, and the first matching rule wins. A workable starting policy logs Secrets, ConfigMaps and token requests at Metadata only (so values never reach the log), RBAC and webhook configuration changes at RequestResponse, and everything else at Metadata. On a managed platform, enable the provider's control plane audit log export instead.
| Signal | Why it is in the first alert set | ATT&CK |
|---|---|---|
Use of the break-glass identity or any system:masters request | Should never happen outside a declared incident | T1078.004 Cloud Accounts |
pods/exec and pods/attach in production namespaces | Interactive access to running workloads | T1609 Container Administration Command |
Privileged pod or hostPath creation attempts | Precursor to node takeover | T1610 Deploy Container, T1611 Escape to Host |
Secret list across namespaces, unusual token requests | Credential harvesting from the API | T1552.007 Container API, T1528 Steal Application Access Token |
| New ClusterRoleBinding or webhook configuration | Persistence and privilege escalation | T1078.004 Cloud Accounts |
| Anonymous or unexpected identities enumerating resources | Discovery before attack | T1613 Container and Resource Discovery |
Audit logs themselves must be protected from general access, as the checklist says, and shipped off the cluster to a store the platform team cannot alter. Query patterns are in the Kubernetes Audit Log Detection Guide.
Gate: each alert above fired once in non-production by a controlled test, received by the SOC and closed with a runbook.
Phase 5: pilot and handover
- Pilot tenant: one team with a real but non-critical service, onboarded through the standard namespace template (quotas, network policy, Pod Security labels, service account, RBAC group bindings).
- Backup and restore: prove restore of cluster state and persistent volumes before the first critical workload. See Kubernetes Backup and Recovery.
- Incident readiness: run a tabletop of the Kubernetes Cluster Compromise Playbook with platform, SOC and the pilot team, including credential rotation for the cluster CA and service account signing keys.
- Health check baseline: take the first reading of the Kubernetes Platform Health Check and agree its cadence with operations.
- Handover: operations sign acceptance of runbooks for upgrades, certificate rotation, encryption key rotation, node patching and break-glass use. Later waves follow only after the pilot has run one minor upgrade or a full patch cycle cleanly.
Where identity fits
A cluster is an identity system in its own right. The API server trusts certificates, bearer tokens, OIDC tokens and service account tokens; controllers, operators, CI pipelines, GitOps agents and monitoring tools all hold credentials that often outlive the people who created them. Treat the CI or GitOps deployer as the most privileged non-human identity on the platform: scope it per namespace, federate it rather than storing a kubeconfig, and review it with the same cadence as human administrators. The OWASP Kubernetes Top 10 (2025) lists K02 Overly Permissive Authorization Configurations, K03 Secrets Management Failures, K08 Cluster To Cloud Lateral Movement and K09 Broken Authentication Mechanisms, four of its ten risks, as identity problems. For stale identities after go-live, use How to find unused Kubernetes service accounts.
Small and large organisations
| Decision | Small team, one or two clusters | Large organisation, fleet |
|---|---|---|
| Control plane | Managed control plane, so etcd and API server patching sit with the provider | Managed by default; self-managed only where residency or air gaps require it, with a dedicated control plane team |
| Tenancy | Cluster per environment, namespaces per application | Cluster per trust tier or business unit, namespaces per team |
| Policy engine | Pod Security Admission plus built-in validating admission policies | Central policy engine with policies distributed from one repository to all clusters |
| Logging | Provider audit export to the existing SIEM | Central pipeline with per-cluster tagging and fleet-wide detections |
| Upgrades | Annual, planned with the provider calendar | Rolling waves, canary clusters first, version skew tracked as a metric |
Detection and measurement
- Standing cluster-admin holders: people with permanent cluster-wide admin. Target zero, excluding the vaulted break-glass identity.
- Enforce coverage: percentage of application namespaces labelled
enforce: restricted. Target 100 per cent with dated exceptions. - Secret encryption: Secrets confirmed encrypted in etcd or by the provider. Target 100 per cent.
- Token automount rate: pods with a service account token mounted that never call the API. Trend downward each wave.
- Version currency: clusters on a minor release the project still maintains. Target 100 per cent.
- Alert test pass rate: first-set detections that fired and reached the SOC in the last quarterly test.
Common failure patterns
- Admin kubeconfig shared on a wiki: the bootstrap credential becomes the team credential, and because it carries
system:mastersno RBAC change can contain it. - Enforce on day one: moving straight to
restrictedbreaks the pilot, the label is removed in a hurry and never comes back. Warn and audit first. - Encryption switched on but never applied: the configuration is in place, the existing Secrets were never rewritten, and the
identityfallback is still there. - Monitoring agent with
nodes/proxy: a broad read role turns out to allow command execution that bypasses audit and admission. - Audit log present, nobody reads it: events are written to the control plane host and rotated away; no alert set was ever tested.
- Exempt the controller: a Pod Security exemption for a controller service account quietly exempts every user who can create a Deployment.
Framework mapping
| Phase | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | CIS Controls v8.1 | ISO/IEC 27001:2022 Annex A |
|---|---|---|---|---|
| 0. Plan | CM-8 | ID.AM-02 | 1, 2 | A.5.9 |
| 1. Baseline | CM-2, CM-6, CM-7, SC-28, SC-12, SI-2 | PR.PS-01, PR.PS-02 | 4, 3, 7 | A.8.9, A.8.24, A.8.8 |
| 2. Identity and RBAC | AC-2, AC-6, AC-6(5), IA-5 | PR.AA-01, PR.AA-05 | 5, 6 | A.5.15, A.5.18, A.8.2, A.8.5 |
| 3. Admission | CM-7, CM-6 | PR.PS-01 | 4, 16 | A.8.9 |
| 4. Observability | AU-2, AU-6, AU-9, AC-6(9) | PR.PS-04, DE.CM-09 | 8, 13 | A.8.15, A.8.16 |
| 5. Handover | CA-7 | DE.CM-09 | 11, 17 | A.5.24, A.8.13 |
Practitioner checklist
- Tenancy model, control plane ownership split and version policy recorded in an architecture decision record.
- All cluster configuration, RBAC, admission and audit policy held in a reviewed repository.
- CIS benchmark scan for the matching distribution attached, exceptions dated and owned.
- Secrets encrypted at rest with a KMS provider, existing Secrets rewritten,
identityfallback removed. system:masterscredential vaulted, tested and alerted on; no standing humancluster-admin.- Escalation permissions (
escalate,bind,impersonate,nodes/proxy, Secretlist) absent from everyday roles. - Every workload on a dedicated service account, automount off where unused, cloud access federated.
- Pod Security Admission in warn and audit from build; enforce
restrictedper namespace when clean. - Audit policy live, logs off-cluster and access-restricted, first alert set tested.
- Restore tested, compromise playbook exercised, health check baseline taken before handover.
Standards and references
- Kubernetes: Security Checklist
- Kubernetes: Role Based Access Control Good Practices
- Kubernetes: Pod Security Admission
- Kubernetes: Auditing
- Kubernetes: Encrypting Confidential Data at Rest
- Kubernetes: Releases and support period
- CIS: Kubernetes Benchmarks
- OWASP: Kubernetes Top 10
Related NHI Mgmt Group resources: Kubernetes Security Guide, Kubernetes Admission Control Guide, Kubernetes Cluster Compromise Playbook, Kubernetes Workload Identity Checklist, Kubernetes Platform Health Check.