Containers are built to be thrown away, so the evidence of a compromise is usually deleted by the platform before an investigator asks for it. A pod that crashes, scales in or is redeployed takes its writable layer, its memory and its local logs with it, and a well-meaning engineer who deletes a suspicious pod has done the attacker's clean-up for them. This guide covers what evidence exists in a containerised workload, where it lives on a Kubernetes node, how to preserve it with checkpoints, pauses and snapshots, and how to prepare clusters so that evidence survives. It is written for incident responders, platform engineers and SOC analysts who already know general forensics and need the container-specific detail. Cloud account and provider-level evidence is covered in the Cloud Forensics Guide; this page goes deeper on the workload and node.
Key takeaways
- Do not delete the pod first. Isolate it with a deny-all NetworkPolicy, cordon the node and stop the controller from replacing it, then capture evidence. Deleting a pod managed by a Deployment simply reschedules the vulnerable workload elsewhere.
- Evidence is split across four layers. The Kubernetes control plane (audit log), the node (kubelet, runtime, journald, disk), the container (writable layer, process memory) and the cloud account (IAM, flow logs). An investigation that reads only one layer misses the story.
- Checkpointing captures memory without alerting the process. The kubelet checkpoint API, beta and enabled by default since Kubernetes v1.30, uses CRIU to write a stateful copy of a running container that can be analysed and restored in a sandbox.
- Checkpoint archives are sensitive data. They hold every memory page of the container, including keys and secrets, so treat them as you would a memory image: restricted storage, hashing and chain of custody.
- Readiness decides the outcome. API server audit logging, central log shipping that outlives kubelet rotation, runtime detection and a pre-approved forensic toolkit must exist before the incident, because none of them can be turned on retrospectively.
Why container forensics differs from host forensics
The forensic process in NIST SP 800-86 (collection, examination, analysis, reporting) still applies, but containers change what can be collected and how quickly it disappears.
| Property | Traditional server | Container on Kubernetes | Forensic consequence |
|---|---|---|---|
| Lifetime | Months or years | Minutes to days; replaced on every deploy | Evidence must be captured in the first response step, not after triage meetings |
| Filesystem | One persistent disk | Read-only image layers plus a thin writable layer, plus mounted volumes | The writable layer is where attacker changes live, and it is discarded with the container |
| Logs | Local syslog with long retention | stdout and stderr files under /var/log/pods, rotated by the kubelet at 10Mi with 5 files by default | A noisy container can overwrite its own history within minutes |
| Identity | Local accounts, domain accounts | Kubernetes service account tokens, cloud workload identity, mounted secrets | Credential theft from a pod leads straight to the API server or cloud account |
| Control | Administrators log in | Changes arrive through the API server, CI/CD or GitOps controllers | The API server audit log is the primary record of who did what |
| Shared kernel | One OS per host | Many containers share the node kernel | A container escape (ATT&CK T1611) turns a pod incident into a node incident |
Where the evidence lives
Map evidence sources before an incident so that responders know what to grab and in what order. The list below follows the order of volatility, most volatile first.
| Layer | Evidence source | How to collect | Survives pod deletion? |
|---|---|---|---|
| Container memory | Process memory, open sockets, decrypted secrets, injected code | Kubelet checkpoint API, docker checkpoint create, node memory capture with LiME | No |
| Container runtime state | Running containers, image digests, mounts, environment, process tree | crictl ps, crictl inspect, nerdctl inspect, docker inspect | No |
| Writable layer | Dropped binaries, modified configuration, web shells, deleted files | docker diff (A added, D deleted, C changed), runtime snapshot directory on the node, docker export | No |
| Container logs | Application stdout and stderr | crictl logs, kubectl logs --previous, files under /var/log/pods | Briefly, until garbage collection |
| Node logs | kubelet and runtime logs, auditd, kernel messages | journalctl -u kubelet, runtime unit logs, auditd files | Yes, until the node is replaced |
| Node disk | Image store, volumes, host files touched after an escape | Cloud volume snapshot or disk image | Yes, until the node is replaced |
| Control plane | API server audit events: exec, create, patch, secret reads, RBAC changes | Audit log backend or the managed provider's control plane logs | Yes, subject to retention |
| Cloud account | IAM role use from the node or pod, flow logs, registry pulls | Provider audit trail and flow logs; see the Cloud Logging Guide | Yes, subject to retention |
Persistent volumes deserve a separate note: docker export does not include the contents of mounted volumes, and when a volume is mounted over an existing directory the export contains the underlying directory instead. Snapshot volumes through the storage layer, not through the container.
First response: isolate without destroying
The AWS EKS best practices guide sets out an order that works on any distribution. Adapt it into a runbook and rehearse it.
- Identify the scope: find the affected pods and their nodes by name, by controller selector, by service account and by image digest. Querying every pod that uses the same compromised service account or image often reveals spread you would otherwise miss.
- Isolate the network: apply a NetworkPolicy that selects the pod by label and denies all ingress and egress. Relabelling the pod to a quarantine label that only the deny policy selects also removes it from Service endpoints. If the attacker may have reached the node itself, a NetworkPolicy is not enough: isolate the node with cloud security groups or host firewall rules, accepting that every container on it is affected.
- Stop the platform from cleaning up: run
kubectl cordonon the node so nothing new is scheduled there, enable instance termination protection (on EC2, theDisableApiTerminationattribute) and autoscaler scale-in protection. Spot instances cannot be protected, so capture their evidence first. Label the pod and node so other administrators know not to touch them. - Revoke credentials: the pod's service account token, its cloud workload identity role and any secrets it mounted are presumed stolen. Rotate them, and remove IAM policies from the node role where that will not break other workloads. See the Kubernetes Service Account Token Guide.
- Capture volatile evidence: checkpoint or pause the container, collect runtime state and logs, then capture node memory if an escape is suspected (next section).
- Snapshot disks: take a snapshot of the node's volumes and any persistent volumes the pod used, and record the snapshot IDs in the case log.
- Redeploy safely: roll out a fixed image or configuration as a new workload first, then delete the compromised one. Deleting a pod managed by a Deployment or DaemonSet only recreates the vulnerable pod.
The full cluster response, including control plane and RBAC clean-up, is in the Kubernetes Cluster Compromise Playbook.
Capturing memory and state: checkpoint, pause and debug
Kubelet checkpoint API
Forensic container checkpointing arrived as alpha in Kubernetes v1.25 and has been beta, behind the ContainerCheckpoint feature gate and enabled by default, since v1.30. It builds on CRIU (Checkpoint/Restore In Userspace), which is integrated into runc, crun, CRI-O and containerd. The key properties for investigators:
- Request:
POST /checkpoint/{namespace}/{pod}/{container}on the kubelet API, with an optionaltimeoutin seconds. Access is controlled by kubelet authentication and authorisation, so grant it only to the forensic role. - Output: a tar archive named
checkpoint-<podFullName>-<containerName>-<timestamp>.tarwritten under the kubelet root directory, by default/var/lib/kubelet/checkpoints. - Stealth: the container is not aware it was checkpointed, and the copy can be restored repeatedly in an isolated sandbox for dynamic analysis.
- Failure modes: a 404 means the feature gate is disabled or the target was not found; a 500 can mean the runtime does not implement the checkpoint CRI call. Test on every node image you run before you need it.
- Duration: checkpoint time grows with the memory the container uses, so set a realistic timeout for large JVM or database containers.
Analyse archives offline with checkpointctl: show and inspect report the image, runtime, network interfaces, process tree and root filesystem diff size; diff compares two checkpoints; memparse dumps memory pages for a given PID. The project's own example finds a database password in process memory, which is exactly why the archive must be handled as sensitive evidence. The Kubernetes documentation notes that runtimes should make the archive readable only by root, and that anyone who later holds a copy can read every memory page.
Pause and other runtimes
- Pause:
docker pauseuses the Linux freezer cgroup, so processes are suspended without receiving a signal they could observe or trap. Pausing stops further damage and freezes state while you collect; it does not preserve anything if the node is later terminated. - Docker checkpoint:
docker checkpoint create(with--leave-runningand--checkpoint-dir) saves container state including memory on hosts where the experimental feature is enabled. - Node memory: when you suspect an escape or kernel-level implant, capture full node memory with LiME and analyse it with Volatility. See the Memory Forensics Guide.
Live inspection with kubectl debug
Ephemeral containers, stable since v1.25, let a responder attach tooling to a running pod with kubectl debug --target=<container> to share its process namespace, which matters for distroless images that have no shell. kubectl debug node/<node> starts a pod in the host IPC, network and PID namespaces with the node root filesystem mounted at /host; --profile=sysadmin gives the privileges needed for packet capture. Be aware of the trade-offs:
- Every action changes the system you are examining, and ephemeral containers cannot be removed from the pod once added. Record each command in the case log.
- Prefer
--copy-towhen you want to experiment, because it creates a copy of the pod rather than altering the original. - Debug access is itself a privileged capability attackers abuse (ATT&CK T1609 Container Administration Command). Restrict the
pods/ephemeralcontainersand node debug permissions to the forensic role and alert on their use.
Control plane and node logs: the record of intent
The API server audit log shows who created, exec'd into, patched or read what, and is often the only evidence that survives a fast attacker. Kubernetes audit policy has four levels: None, Metadata (user, timestamp, resource, verb), Request (adds the request body) and RequestResponse (adds the response body). On self-managed clusters configure the log backend with --audit-log-path and retention with --audit-log-maxage, then ship the files off the control plane. On managed services, enable the provider's control plane audit logs, which are usually off or partial by default.
- Log at Metadata for everything: it is cheap and answers most "who did it" questions.
- Raise the level for high-value resources:
pods/exec,pods/attach,pods/ephemeralcontainers, RBAC objects, admission webhooks and workload creation in sensitive namespaces. - Never log Secret bodies: keep Secrets, ConfigMaps that hold credentials and token requests at
Metadataso the audit log does not become a secret store. - Ship node logs too: the kubelet and runtime write to journald on systemd nodes (
journalctl -u kubelet), and container log files under/var/log/podsrotate by size. A central pipeline is the only way to keep history beyondcontainerLogMaxSizeandcontainerLogMaxFiles.
Detection queries for these logs are covered in the Kubernetes Audit Log Detection Guide.
Analysis: reconstructing what happened
| Question | Evidence to examine | ATT&CK technique to test |
|---|---|---|
| How did the attacker get code running? | Audit log create and exec events; image digests versus the approved registry | T1610 Deploy Container, T1609 Container Administration Command |
| Was the image itself poisoned? | Image digest history, registry push logs, SBOM and signature verification | T1525 Implant Internal Image, T1612 Build Image on Host |
| What did they look for? | API discovery calls in the audit log, process tree in the checkpoint | T1613 Container and Resource Discovery |
| Which credentials were exposed? | Mounted secrets, environment variables in the checkpoint, service account token use from new source IPs | T1552.007 Container API |
| Did they reach the node? | Privileged pod specs, hostPath mounts, node memory and disk, kernel module loads | T1611 Escape to Host |
| Did they cover tracks? | Gaps in log sequence, deleted pods, changes to logging agents or runtime security sensors | T1070 Indicator Removal, T1685 Disable or Modify Tools |
Build a single timeline that merges audit events, runtime alerts, node journal entries and cloud audit trail records, normalised to UTC. Kubernetes object UIDs and container IDs join the layers: the audit log records the pod UID, the runtime records the container ID, and the checkpoint metadata records both. The Forensic Timeline Analysis Guide covers the general method.
Preparing clusters for forensics
- Runtime detection: deploy an eBPF or syscall-based sensor. The open source Falco default rules include, for example, "Terminal shell in container", "Contact K8S API Server From Container", "Drop and execute new binary in container", "Fileless execution via memfd_create", "Detect release_agent File Container Escapes" and "Clear Log Activities". These alerts mark the moment to start capture. See the Container Runtime Security Guide.
- Immutable workloads: run with
readOnlyRootFilesystem: truewhere possible, so anything written outside declared volumes is an anomaly, and pin images by digest so the image in evidence matches the image in the registry. - Forensic role and toolkit: a break-glass RBAC role with checkpoint, debug and log read rights; an approved debug image with known hashes; and storage for evidence that the cluster's own workloads cannot write to.
- Evidence handling: hash every archive and snapshot at collection, record who collected it and when, and copy it to write-once storage outside the affected account. The Forensic Readiness Guide covers chain of custody.
- Retention that outlives the pod: keep audit and container logs centrally for at least as long as your typical detection gap, and test that a pod deleted a week ago can still be reconstructed.
- Exercises: run a drill where a responder must checkpoint a container, snapshot its node and produce a timeline without deleting anything. Most teams discover a missing permission or a disabled feature gate the first time.
Where identity fits
Most container incidents become identity incidents within minutes, because the first thing an attacker does in a pod is read its service account token, cloud credentials and mounted secrets. Investigators should treat every non-human identity the workload could reach as in scope: the Kubernetes service account, the cloud role bound through workload identity, registry pull credentials and any API keys in environment variables. Check where each was used after the compromise time, not only where it was supposed to be used. See the Kubernetes NHI Security Guide.
Common failure patterns
- Deleting the pod to "contain" it: the controller recreates it, the writable layer and memory are lost, and the vulnerability is still live.
- No API server audit log: many managed clusters ship without control plane logging enabled, so the question "who ran exec in that pod" has no answer.
- Relying on kubelet log rotation: with default rotation a busy container keeps only a few tens of megabytes of history, which a cryptominer's output can exhaust quickly.
- Ignoring the node and cloud layers: in the Tesla Kubernetes cryptojacking incident the attackers found AWS credentials in a pod, and the duration of access was never established. Pod evidence alone cannot answer what was done with stolen cloud keys.
- Not hunting by image and identity: in the Kubeflow cryptomining attacks the same suspicious image appeared across many clusters and abused a mounted service account. Scoping by image digest and service account finds siblings that scoping by pod name misses.
- Mishandling checkpoint archives: leaving them on the node, copying them to shared buckets or attaching them to tickets exposes every secret the container held.
Framework mapping
| Framework | Reference | What it asks for here |
|---|---|---|
| NIST CSF 2.0 | RS.AN-03, RS.AN-06, RS.AN-07, DE.AE-02 | Root cause analysis; recorded investigation actions; incident data collected with integrity and provenance preserved; analysis of adverse events |
| NIST SP 800-53 Rev. 5 | AU-2, AU-9, AU-11, AU-12, IR-4, SI-4 | Audit event selection, protection and retention; incident handling; system monitoring |
| NIST SP 800-86 | Collection, examination, analysis, reporting | The forensic process applied to container data sources |
| CIS Controls v8.1 | Control 8, Control 17 | Audit log management; incident response management |
| ISO/IEC 27001:2022 Annex A | A.5.28, A.8.15 | Collection of evidence; logging |
| MITRE ATT&CK v20 | T1609, T1610, T1611, T1613, T1525, T1552.007 | Container techniques to test hypotheses against |
Practitioner checklist
- Enable API server audit logging at Metadata level for all requests and higher levels for exec, attach, ephemeral containers and RBAC changes, shipped off-cluster.
- Ship container, kubelet and runtime logs centrally with retention longer than your detection gap.
- Confirm the
ContainerCheckpointfeature and your runtime's checkpoint support work on every node image, and record the procedure. - Create a forensic RBAC role for checkpoint, debug and log access, and alert whenever it is used.
- Write a first-response runbook: identify by image and service account, isolate with NetworkPolicy, cordon, protect from termination, revoke credentials, capture, snapshot, then redeploy.
- Prepare a hashed debug image,
checkpointctland memory capture tooling before any incident. - Store evidence in write-once storage outside the affected account, with hashes and a custody log.
- Deploy runtime detection and wire its alerts to automatic evidence capture where safe.
- Rehearse the runbook at least twice a year and fix what the drill exposes.
Standards and references
- NIST: SP 800-86 Guide to Integrating Forensic Techniques into Incident Response
- Kubernetes: Kubelet Checkpoint API
- Kubernetes: Forensic container checkpointing in Kubernetes
- Kubernetes: Auditing
- Kubernetes: Debug Running Pods
- AWS: EKS Best Practices, Incident response and forensics
- CRIU project: checkpointctl
- MITRE ATT&CK: Escape to Host
Related NHI Mgmt Group resources: Cloud Forensics Guide · Kubernetes Cluster Compromise Playbook · Digital Forensics (DFIR) Guide · Kubernetes Security Guide