When an attacker gets into a Kubernetes cluster, the usual incident steps of isolating a host and resetting a password do not map cleanly onto pods that reschedule themselves, tokens that are minted per pod and RBAC bindings that quietly grant cluster-wide power. Most cluster compromises start with an exposed dashboard, API server or kubelet, a vulnerable workload, or a stolen kubeconfig, and then turn on a service account token with more rights than anyone realised. This template is a playbook for responding to a compromise of a Kubernetes cluster, its nodes or its workload identities. It covers cluster-specific containment (cordon, network policy, RBAC and admission control), token invalidation, evidence capture before pods disappear, and rebuilding. It is written for platform, security operations and cloud teams. Adapt roles and tool names to [Organisation] and to your distribution (self-managed, EKS, AKS, GKE or others).
How to use this template
- Replace text in square brackets, such as [Organisation], [Cluster owner] and [Incident manager], with your names and contact routes, then approve it through your policy process.
- Steps marked Decision: need a recorded decision, decision maker and time.
- Commands are named, not scripted. Test each containment action in a non-production cluster first and record the exact command your platform uses.
- This playbook goes deeper than the Kubernetes Security Guide, which covers prevention. Use it with the Kubernetes Audit Log Detection Guide for detection logic and the Container and Kubernetes Forensics Guide for evidence handling.
- It follows the life cycle in NIST SP 800-61 Rev. 3 (a CSF 2.0 Community Profile).
1. Scope and triggers
In scope: the control plane (API server, etcd, controller manager, scheduler), worker nodes and kubelets, workloads and container images, service accounts and their tokens, human kubeconfigs and cloud IAM bindings to the cluster, admission controllers and add-ons such as dashboards, notebooks and CI agents running in the cluster.
| Trigger | Typical signal | ATT&CK |
|---|---|---|
| Exposed management interface | Dashboard, API server or kubelet reachable from the internet; anonymous requests (system:anonymous) succeeding | T1133 External Remote Services, T1190 Exploit Public-Facing Application |
| Service account token abuse | A service account calling verbs or resources it has never used: listing secrets, creating pods, creating role bindings | T1078.001 Default Accounts, T1528 Steal Application Access Token |
| Unexpected workloads | New pods, DaemonSets or CronJobs from unknown images; cryptomining CPU patterns | T1610 Deploy Container, T1053.007 Container Orchestration Job, T1496.001 Compute Hijacking |
| Container escape | Privileged or hostPath pods created; host processes spawned from a container runtime; node file changes | T1611 Escape to Host |
| Interactive access to workloads | pods/exec, pods/attach or kubelet exec calls from unusual identities | T1609 Container Administration Command |
| Credential discovery | Reads of secrets across namespaces; pod requests to the cloud metadata address 169.254.169.254 | T1552.007 Container API, T1552.005 Cloud Instance Metadata API |
| Poisoned image | Image digest in a registry changed, or an image not built by your pipeline running in production | T1525 Implant Internal Image, T1204.003 Malicious Image |
Severity: Critical if the attacker has cluster-admin, can write to kube-system, has reached a node as root, or holds credentials for etcd or cloud IAM. High if limited to one namespace with secret read access. Medium for a single workload with no evidence of token use.
2. Roles and contacts
| Role | Responsibilities | Contact |
|---|---|---|
| [Incident manager] | Declares, sets severity, owns decisions and timeline | [Name, out-of-hours route] |
| [Cluster owner or platform on-call] | Runs containment commands, holds break-glass cluster-admin, rebuilds nodes | [On-call rota] |
| [SOC and DFIR] | Audit log analysis, node and container evidence capture, hunting | [SOC queue] |
| [Cloud team] | Cloud IAM roles bound to the cluster and nodes, cloud audit logs, node instance isolation | [Team] |
| [Application owners] | Confirm normal behaviour of their workloads, accept downtime decisions | [Service catalogue] |
| [Secret owners and NHI team] | Rotate every secret and credential reachable from the cluster | From the secrets inventory |
| [Legal, privacy and communications] | Notification decisions | [Contacts] |
Hold the break-glass credential out of band. If the attacker can modify RBAC, your normal admin route may already be watched or altered.
3. Detection and triage
- Confirm audit logging exists. Kubernetes audit events are generated by kube-apiserver only when an audit policy is configured with
--audit-policy-file; if the flag is omitted, no events are logged. On managed services, confirm control plane audit logs are enabled and exported. If they are not, enable them now and record the gap. - Identify the acting identity. From audit events (
user.username,user.groups,sourceIPs,userAgent), determine whether activity came from a human kubeconfig, a cloud IAM identity mapped into the cluster, a service account (system:serviceaccount:<namespace>:<name>) or anonymous access. - Map what that identity can do. List its RoleBindings and ClusterRoleBindings and check for the high-risk rights:
secretsget or list,podscreate,pods/exec,escalate,bind,impersonate, and anything bound tocluster-admin. See the Kubernetes RBAC Guide. - Hunt for persistence: new ClusterRoleBindings, service accounts, long-lived token Secrets (
type: kubernetes.io/service-account-token), DaemonSets, CronJobs, static pod manifests on nodes, mutating admission webhooks and changes to image references. - Check for escape. Look for pods with
privileged: true,hostPID,hostNetworkor hostPath mounts of/or the container runtime socket, created in the window. Treat any node that ran one as compromised. - Check the cloud boundary. Review cloud audit logs for use of node instance roles and workload identity roles from outside the cluster.
- Preserve evidence before pods vanish: export audit logs, snapshot node disks, capture container filesystem and logs from suspect pods, and record pod specs and image digests. Controllers may replace a deleted pod within seconds, destroying evidence.
- Decision: severity and whether to treat the whole cluster as untrusted (rebuild) or contain in place.
4. Containment
Contain from the outside in: cut external access, then identities, then workloads, then nodes. Record every change so it can be reversed.
| Action | Effect | Watch out for |
|---|---|---|
| Restrict API server access to admin networks; remove public LoadBalancers and ExternalIPs on dashboards | Closes the entry point | CI systems that call the API from the internet |
Disable kubelet anonymous access (--anonymous-auth=false) and use --authorization-mode=Webhook | Kubelet API requires authenticated, authorised requests | The kubelet default authorisation mode is AlwaysAllow |
| Delete attacker-created RoleBindings and ClusterRoleBindings; strip excess rights from abused service accounts | Removes the attacker's API power | Capture the objects as evidence before deleting |
| Delete pods holding suspect projected tokens; delete legacy token Secrets | Kubernetes invalidates tokens bound to a deleted Pod or Secret immediately | There is no other way to revoke a single TokenRequest token; stolen copies stay valid until the object is deleted or the token expires |
| Apply a deny-all NetworkPolicy to the affected namespace or label the suspect pods into one | Isolates the pod while keeping it for evidence | Only works if your CNI enforces NetworkPolicy |
kubectl cordon the affected node | Marks the node unschedulable; running pods keep running | Use drain only after evidence capture, because it evicts pods |
| Isolate the node at the cloud or network layer | Stops a node compromised by escape from reaching other systems | Kubelet loses contact; workloads reschedule elsewhere |
Enforce Pod Security Admission restricted (or at least baseline) on namespaces | Blocks new privileged pods used for escape | Run in warn or audit mode first if workloads may break |
| Block the cloud metadata address from pods | Stops harvesting of node instance credentials | Workloads relying on node roles |
- Decision: keep suspect pods running in isolation for evidence, or delete them to invalidate tokens. Where tokens are actively being abused, invalidation wins.
- Revoke human kubeconfig credentials and cloud IAM sessions for any admin identity seen in the attack.
5. Eradication
- Rebuild compromised nodes from a clean image rather than cleaning them. If an attacker had root on a node, assume the kubelet credentials and every secret mounted on that node are exposed.
- Remove all persistence found in section 3, including admission webhooks and static pod manifests.
- Rotate secrets readable by the attacker's identities: Kubernetes Secrets, registry pull secrets, database and API credentials, cloud keys stored in pods. See the Kubernetes Secrets Security Guide.
- Replace long-lived service account tokens. Legacy Secret-based tokens do not expire. Move workloads to projected tokens, which the kubelet rotates and which expire automatically, and set
automountServiceAccountToken: falsewhere a workload does not call the API. See the Kubernetes Service Account Token Guide. - If cluster-admin or etcd was reached: rotate the cluster certificate authority and service account signing keys, or rebuild the cluster and migrate workloads. Decision: rotate in place or rebuild; for most organisations rebuilding from infrastructure as code is quicker and more certain.
- Rebuild images from source with your pipeline and redeploy by digest.
6. Recovery
- Restore workloads namespace by namespace, starting with the highest business priority, using images and manifests from source control.
- Uncordon or replace nodes only after admission policy, network policy and RBAC changes are in place.
- Monitor for at least [30] days: failed authentications with old tokens, new ClusterRoleBindings, privileged pod creation and egress to mining pools or unknown hosts.
- Decision: close when nodes are rebuilt, all reachable secrets rotated, no persistence remains, and audit logging is confirmed working.
7. Communication and notification
| Audience | Owner | When | Content |
|---|---|---|---|
| Application owners | [Cluster owner] | Before any drain, network isolation or deletion | Expected impact, timing, rollback |
| Executives | [Incident manager] | At declaration and daily | Scope, data at risk, service impact |
| Cloud provider | [Cloud team] | If the control plane is managed and its integrity is in doubt | Support case with timeline and evidence |
| Customers and regulators | [Legal and privacy] | Per contract and law | If personal or customer data in the cluster was accessible |
8. Post-incident review
- Hold the review within [10] working days. Build a timeline from first access through token abuse to containment.
- Ask: were audit logs enabled and retained long enough? Which service account had more rights than its workload needed? Was anything exposed that the security checklist says should never be?
- Record actions in the backlog with owners and dates, and update this playbook.
Implementation guidance
Prepare before the incident. The Kubernetes security checklist says the API server, kubelet API and etcd should not be exposed publicly, and that LoadBalancer and ExternalIPs use should be restricted. Beyond that:
- Audit policy: log at least
Metadatafor all requests, andRequestResponsefor RBAC objects and pod creation; ship to storage outside the cluster. - Workload identity: remove legacy token Secrets. From Kubernetes 1.29, unused auto-generated legacy tokens are invalidated after a period (by default one year) and later purged, and the
kubernetes.io/legacy-token-last-usedlabel shows which are still in use. - Admission: Pod Security Admission labels on every namespace, image provenance or registry allow-lists, and the
NodeRestrictionadmission plugin. See the Kubernetes Admission Control Guide. - Rehearsal: test cordon, network isolation, pod deletion and node replacement in a staging cluster, and time them.
Small vs large organisations. A small team with one or two managed clusters should favour rebuild over forensic depth: keep everything in infrastructure as code and practise recreating a cluster. A large organisation with many clusters needs central audit log collection, a fleet view of RBAC bindings and a standard containment runbook per distribution; the hard part is usually finding which clusters trust a stolen identity.
| Framework | Mapping |
|---|---|
| NIST CSF 2.0 | DE.CM-09, DE.AE-02, RS.MA-01, RS.AN-03, RS.MI-01, RS.MI-02, RC.RP-01 |
| NIST SP 800-53 Rev. 5 | IR-4 Incident Handling, IR-5 Incident Monitoring, AU-2 Event Logging, AU-12 Audit Record Generation, AC-6 Least Privilege, CM-2 Baseline Configuration, SC-7 Boundary Protection |
| MITRE ATT&CK | T1133, T1190, T1078.001, T1528, T1609, T1610, T1611, T1053.007, T1552.007, T1552.005, T1525, T1204.003, T1496.001 |
Readiness metrics: percentage of clusters with audit logs exported off-cluster; time to list every binding of a named identity across all clusters; number of legacy service account token Secrets remaining; percentage of namespaces enforcing Pod Security baseline or stricter; time to replace a node in a rehearsal.
Common failure patterns:
- Exposed consoles with mounted credentials: the Tesla Kubernetes cryptojacking case began with an unauthenticated console and AWS keys inside a pod.
- Default service accounts with deploy rights: in the Kubeflow cryptomining attacks, exposed dashboards let attackers use a notebook's mounted service account to deploy miners.
- Deleting a suspect pod before capturing it, then watching the controller recreate it with the evidence gone.
- Rotating Kubernetes Secrets but leaving node instance roles and cloud keys untouched.
- Cleaning a node after an escape instead of replacing it.
References: NIST: SP 800-61 Rev. 3 Incident Response Recommendations · Kubernetes: Auditing · Kubernetes: Managing Service Accounts · Kubernetes: Service Accounts · Kubernetes: Kubelet authentication and authorization · Kubernetes: Security Checklist
Version history: v1.0 [date], approved by [Approver]. Review every [12] months or after each incident.
Related NHI Mgmt Group resources: Kubernetes Security Guide · Kubernetes NHI Security Guide · Container and Kubernetes Forensics Guide · Cloud Incident Response Guide