Join our Newsletter — 33% off our NHI Course
Home› Guides› Container and Kubernetes Forensics Guide: Capture Evidence Before…
Guide Incident Response & Forensics

Container and Kubernetes Forensics Guide: Capture Evidence Before Ephemeral Workloads Disappear

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 15 min read
On this page

Containers are built to be thrown away, so the evidence of a compromise is usually deleted by the platform before an investigator asks for it. A pod that crashes, scales in or is redeployed takes its writable layer, its memory and its local logs with it, and a well-meaning engineer who deletes a suspicious pod has done the attacker's clean-up for them. This guide covers what evidence exists in a containerised workload, where it lives on a Kubernetes node, how to preserve it with checkpoints, pauses and snapshots, and how to prepare clusters so that evidence survives. It is written for incident responders, platform engineers and SOC analysts who already know general forensics and need the container-specific detail. Cloud account and provider-level evidence is covered in the Cloud Forensics Guide; this page goes deeper on the workload and node.

Key takeaways

  • Do not delete the pod first. Isolate it with a deny-all NetworkPolicy, cordon the node and stop the controller from replacing it, then capture evidence. Deleting a pod managed by a Deployment simply reschedules the vulnerable workload elsewhere.
  • Evidence is split across four layers. The Kubernetes control plane (audit log), the node (kubelet, runtime, journald, disk), the container (writable layer, process memory) and the cloud account (IAM, flow logs). An investigation that reads only one layer misses the story.
  • Checkpointing captures memory without alerting the process. The kubelet checkpoint API, beta and enabled by default since Kubernetes v1.30, uses CRIU to write a stateful copy of a running container that can be analysed and restored in a sandbox.
  • Checkpoint archives are sensitive data. They hold every memory page of the container, including keys and secrets, so treat them as you would a memory image: restricted storage, hashing and chain of custody.
  • Readiness decides the outcome. API server audit logging, central log shipping that outlives kubelet rotation, runtime detection and a pre-approved forensic toolkit must exist before the incident, because none of them can be turned on retrospectively.

Why container forensics differs from host forensics

The forensic process in NIST SP 800-86 (collection, examination, analysis, reporting) still applies, but containers change what can be collected and how quickly it disappears.

PropertyTraditional serverContainer on KubernetesForensic consequence
LifetimeMonths or yearsMinutes to days; replaced on every deployEvidence must be captured in the first response step, not after triage meetings
FilesystemOne persistent diskRead-only image layers plus a thin writable layer, plus mounted volumesThe writable layer is where attacker changes live, and it is discarded with the container
LogsLocal syslog with long retentionstdout and stderr files under /var/log/pods, rotated by the kubelet at 10Mi with 5 files by defaultA noisy container can overwrite its own history within minutes
IdentityLocal accounts, domain accountsKubernetes service account tokens, cloud workload identity, mounted secretsCredential theft from a pod leads straight to the API server or cloud account
ControlAdministrators log inChanges arrive through the API server, CI/CD or GitOps controllersThe API server audit log is the primary record of who did what
Shared kernelOne OS per hostMany containers share the node kernelA container escape (ATT&CK T1611) turns a pod incident into a node incident

Where the evidence lives

Map evidence sources before an incident so that responders know what to grab and in what order. The list below follows the order of volatility, most volatile first.

LayerEvidence sourceHow to collectSurvives pod deletion?
Container memoryProcess memory, open sockets, decrypted secrets, injected codeKubelet checkpoint API, docker checkpoint create, node memory capture with LiMENo
Container runtime stateRunning containers, image digests, mounts, environment, process treecrictl ps, crictl inspect, nerdctl inspect, docker inspectNo
Writable layerDropped binaries, modified configuration, web shells, deleted filesdocker diff (A added, D deleted, C changed), runtime snapshot directory on the node, docker exportNo
Container logsApplication stdout and stderrcrictl logs, kubectl logs --previous, files under /var/log/podsBriefly, until garbage collection
Node logskubelet and runtime logs, auditd, kernel messagesjournalctl -u kubelet, runtime unit logs, auditd filesYes, until the node is replaced
Node diskImage store, volumes, host files touched after an escapeCloud volume snapshot or disk imageYes, until the node is replaced
Control planeAPI server audit events: exec, create, patch, secret reads, RBAC changesAudit log backend or the managed provider's control plane logsYes, subject to retention
Cloud accountIAM role use from the node or pod, flow logs, registry pullsProvider audit trail and flow logs; see the Cloud Logging GuideYes, subject to retention

Persistent volumes deserve a separate note: docker export does not include the contents of mounted volumes, and when a volume is mounted over an existing directory the export contains the underlying directory instead. Snapshot volumes through the storage layer, not through the container.

First response: isolate without destroying

The AWS EKS best practices guide sets out an order that works on any distribution. Adapt it into a runbook and rehearse it.

  1. Identify the scope: find the affected pods and their nodes by name, by controller selector, by service account and by image digest. Querying every pod that uses the same compromised service account or image often reveals spread you would otherwise miss.
  2. Isolate the network: apply a NetworkPolicy that selects the pod by label and denies all ingress and egress. Relabelling the pod to a quarantine label that only the deny policy selects also removes it from Service endpoints. If the attacker may have reached the node itself, a NetworkPolicy is not enough: isolate the node with cloud security groups or host firewall rules, accepting that every container on it is affected.
  3. Stop the platform from cleaning up: run kubectl cordon on the node so nothing new is scheduled there, enable instance termination protection (on EC2, the DisableApiTermination attribute) and autoscaler scale-in protection. Spot instances cannot be protected, so capture their evidence first. Label the pod and node so other administrators know not to touch them.
  4. Revoke credentials: the pod's service account token, its cloud workload identity role and any secrets it mounted are presumed stolen. Rotate them, and remove IAM policies from the node role where that will not break other workloads. See the Kubernetes Service Account Token Guide.
  5. Capture volatile evidence: checkpoint or pause the container, collect runtime state and logs, then capture node memory if an escape is suspected (next section).
  6. Snapshot disks: take a snapshot of the node's volumes and any persistent volumes the pod used, and record the snapshot IDs in the case log.
  7. Redeploy safely: roll out a fixed image or configuration as a new workload first, then delete the compromised one. Deleting a pod managed by a Deployment or DaemonSet only recreates the vulnerable pod.

The full cluster response, including control plane and RBAC clean-up, is in the Kubernetes Cluster Compromise Playbook.

Capturing memory and state: checkpoint, pause and debug

Kubelet checkpoint API

Forensic container checkpointing arrived as alpha in Kubernetes v1.25 and has been beta, behind the ContainerCheckpoint feature gate and enabled by default, since v1.30. It builds on CRIU (Checkpoint/Restore In Userspace), which is integrated into runc, crun, CRI-O and containerd. The key properties for investigators:

  • Request: POST /checkpoint/{namespace}/{pod}/{container} on the kubelet API, with an optional timeout in seconds. Access is controlled by kubelet authentication and authorisation, so grant it only to the forensic role.
  • Output: a tar archive named checkpoint-<podFullName>-<containerName>-<timestamp>.tar written under the kubelet root directory, by default /var/lib/kubelet/checkpoints.
  • Stealth: the container is not aware it was checkpointed, and the copy can be restored repeatedly in an isolated sandbox for dynamic analysis.
  • Failure modes: a 404 means the feature gate is disabled or the target was not found; a 500 can mean the runtime does not implement the checkpoint CRI call. Test on every node image you run before you need it.
  • Duration: checkpoint time grows with the memory the container uses, so set a realistic timeout for large JVM or database containers.

Analyse archives offline with checkpointctl: show and inspect report the image, runtime, network interfaces, process tree and root filesystem diff size; diff compares two checkpoints; memparse dumps memory pages for a given PID. The project's own example finds a database password in process memory, which is exactly why the archive must be handled as sensitive evidence. The Kubernetes documentation notes that runtimes should make the archive readable only by root, and that anyone who later holds a copy can read every memory page.

Pause and other runtimes

  • Pause: docker pause uses the Linux freezer cgroup, so processes are suspended without receiving a signal they could observe or trap. Pausing stops further damage and freezes state while you collect; it does not preserve anything if the node is later terminated.
  • Docker checkpoint: docker checkpoint create (with --leave-running and --checkpoint-dir) saves container state including memory on hosts where the experimental feature is enabled.
  • Node memory: when you suspect an escape or kernel-level implant, capture full node memory with LiME and analyse it with Volatility. See the Memory Forensics Guide.

Live inspection with kubectl debug

Ephemeral containers, stable since v1.25, let a responder attach tooling to a running pod with kubectl debug --target=<container> to share its process namespace, which matters for distroless images that have no shell. kubectl debug node/<node> starts a pod in the host IPC, network and PID namespaces with the node root filesystem mounted at /host; --profile=sysadmin gives the privileges needed for packet capture. Be aware of the trade-offs:

  • Every action changes the system you are examining, and ephemeral containers cannot be removed from the pod once added. Record each command in the case log.
  • Prefer --copy-to when you want to experiment, because it creates a copy of the pod rather than altering the original.
  • Debug access is itself a privileged capability attackers abuse (ATT&CK T1609 Container Administration Command). Restrict the pods/ephemeralcontainers and node debug permissions to the forensic role and alert on their use.

Control plane and node logs: the record of intent

The API server audit log shows who created, exec'd into, patched or read what, and is often the only evidence that survives a fast attacker. Kubernetes audit policy has four levels: None, Metadata (user, timestamp, resource, verb), Request (adds the request body) and RequestResponse (adds the response body). On self-managed clusters configure the log backend with --audit-log-path and retention with --audit-log-maxage, then ship the files off the control plane. On managed services, enable the provider's control plane audit logs, which are usually off or partial by default.

  • Log at Metadata for everything: it is cheap and answers most "who did it" questions.
  • Raise the level for high-value resources: pods/exec, pods/attach, pods/ephemeralcontainers, RBAC objects, admission webhooks and workload creation in sensitive namespaces.
  • Never log Secret bodies: keep Secrets, ConfigMaps that hold credentials and token requests at Metadata so the audit log does not become a secret store.
  • Ship node logs too: the kubelet and runtime write to journald on systemd nodes (journalctl -u kubelet), and container log files under /var/log/pods rotate by size. A central pipeline is the only way to keep history beyond containerLogMaxSize and containerLogMaxFiles.

Detection queries for these logs are covered in the Kubernetes Audit Log Detection Guide.

Analysis: reconstructing what happened

QuestionEvidence to examineATT&CK technique to test
How did the attacker get code running?Audit log create and exec events; image digests versus the approved registryT1610 Deploy Container, T1609 Container Administration Command
Was the image itself poisoned?Image digest history, registry push logs, SBOM and signature verificationT1525 Implant Internal Image, T1612 Build Image on Host
What did they look for?API discovery calls in the audit log, process tree in the checkpointT1613 Container and Resource Discovery
Which credentials were exposed?Mounted secrets, environment variables in the checkpoint, service account token use from new source IPsT1552.007 Container API
Did they reach the node?Privileged pod specs, hostPath mounts, node memory and disk, kernel module loadsT1611 Escape to Host
Did they cover tracks?Gaps in log sequence, deleted pods, changes to logging agents or runtime security sensorsT1070 Indicator Removal, T1685 Disable or Modify Tools

Build a single timeline that merges audit events, runtime alerts, node journal entries and cloud audit trail records, normalised to UTC. Kubernetes object UIDs and container IDs join the layers: the audit log records the pod UID, the runtime records the container ID, and the checkpoint metadata records both. The Forensic Timeline Analysis Guide covers the general method.

Preparing clusters for forensics

  • Runtime detection: deploy an eBPF or syscall-based sensor. The open source Falco default rules include, for example, "Terminal shell in container", "Contact K8S API Server From Container", "Drop and execute new binary in container", "Fileless execution via memfd_create", "Detect release_agent File Container Escapes" and "Clear Log Activities". These alerts mark the moment to start capture. See the Container Runtime Security Guide.
  • Immutable workloads: run with readOnlyRootFilesystem: true where possible, so anything written outside declared volumes is an anomaly, and pin images by digest so the image in evidence matches the image in the registry.
  • Forensic role and toolkit: a break-glass RBAC role with checkpoint, debug and log read rights; an approved debug image with known hashes; and storage for evidence that the cluster's own workloads cannot write to.
  • Evidence handling: hash every archive and snapshot at collection, record who collected it and when, and copy it to write-once storage outside the affected account. The Forensic Readiness Guide covers chain of custody.
  • Retention that outlives the pod: keep audit and container logs centrally for at least as long as your typical detection gap, and test that a pod deleted a week ago can still be reconstructed.
  • Exercises: run a drill where a responder must checkpoint a container, snapshot its node and produce a timeline without deleting anything. Most teams discover a missing permission or a disabled feature gate the first time.

Where identity fits

Most container incidents become identity incidents within minutes, because the first thing an attacker does in a pod is read its service account token, cloud credentials and mounted secrets. Investigators should treat every non-human identity the workload could reach as in scope: the Kubernetes service account, the cloud role bound through workload identity, registry pull credentials and any API keys in environment variables. Check where each was used after the compromise time, not only where it was supposed to be used. See the Kubernetes NHI Security Guide.

Common failure patterns

  • Deleting the pod to "contain" it: the controller recreates it, the writable layer and memory are lost, and the vulnerability is still live.
  • No API server audit log: many managed clusters ship without control plane logging enabled, so the question "who ran exec in that pod" has no answer.
  • Relying on kubelet log rotation: with default rotation a busy container keeps only a few tens of megabytes of history, which a cryptominer's output can exhaust quickly.
  • Ignoring the node and cloud layers: in the Tesla Kubernetes cryptojacking incident the attackers found AWS credentials in a pod, and the duration of access was never established. Pod evidence alone cannot answer what was done with stolen cloud keys.
  • Not hunting by image and identity: in the Kubeflow cryptomining attacks the same suspicious image appeared across many clusters and abused a mounted service account. Scoping by image digest and service account finds siblings that scoping by pod name misses.
  • Mishandling checkpoint archives: leaving them on the node, copying them to shared buckets or attaching them to tickets exposes every secret the container held.

Framework mapping

FrameworkReferenceWhat it asks for here
NIST CSF 2.0RS.AN-03, RS.AN-06, RS.AN-07, DE.AE-02Root cause analysis; recorded investigation actions; incident data collected with integrity and provenance preserved; analysis of adverse events
NIST SP 800-53 Rev. 5AU-2, AU-9, AU-11, AU-12, IR-4, SI-4Audit event selection, protection and retention; incident handling; system monitoring
NIST SP 800-86Collection, examination, analysis, reportingThe forensic process applied to container data sources
CIS Controls v8.1Control 8, Control 17Audit log management; incident response management
ISO/IEC 27001:2022 Annex AA.5.28, A.8.15Collection of evidence; logging
MITRE ATT&CK v20T1609, T1610, T1611, T1613, T1525, T1552.007Container techniques to test hypotheses against

Practitioner checklist

  • Enable API server audit logging at Metadata level for all requests and higher levels for exec, attach, ephemeral containers and RBAC changes, shipped off-cluster.
  • Ship container, kubelet and runtime logs centrally with retention longer than your detection gap.
  • Confirm the ContainerCheckpoint feature and your runtime's checkpoint support work on every node image, and record the procedure.
  • Create a forensic RBAC role for checkpoint, debug and log access, and alert whenever it is used.
  • Write a first-response runbook: identify by image and service account, isolate with NetworkPolicy, cordon, protect from termination, revoke credentials, capture, snapshot, then redeploy.
  • Prepare a hashed debug image, checkpointctl and memory capture tooling before any incident.
  • Store evidence in write-once storage outside the affected account, with hashes and a custody log.
  • Deploy runtime detection and wire its alerts to automatic evidence capture where safe.
  • Rehearse the runbook at least twice a year and fix what the drill exposes.

Standards and references

Related NHI Mgmt Group resources: Cloud Forensics Guide · Kubernetes Cluster Compromise Playbook · Digital Forensics (DFIR) Guide · Kubernetes Security Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.