Join our Newsletter — 33% off our NHI Course
Home› Guides› Container Runtime Security Guide: Detecting and Containing Threats…
Guide Cloud Security

Container Runtime Security Guide: Detecting and Containing Threats in Running Workloads

PDF for newsletter subscribers · free to read online

← All guides
By Lalit Choda, NHI Mgmt Group Updated 24 October 2026 17 min read
On this page

Image scanning and admission policy decide what is allowed to start, but nothing in that pipeline tells you what a container does once it is running. An attacker who lands in a pod through a vulnerable web framework, an exposed dashboard or a stolen service account token works inside a process that passed every pre-deployment check. Runtime security closes that gap: it restricts what a running container can do at the kernel level and watches what it actually does, so that a shell, a dropped binary or an escape attempt is blocked or seen within seconds. This guide covers the runtime layer for containers and Kubernetes: kernel hardening with seccomp, AppArmor and SELinux, sandboxed runtimes, eBPF sensors such as Falco and Tetragon, detection rules, response and measurement. It is written for platform engineers, detection engineers and SOC teams who run containers in production. Host Linux detection (auditd, EDR on the node) is covered in the Linux Detection and Response Guide, and the wider container lifecycle in the Container Security Guide.

Key takeaways

  • Containers are processes, not boundaries. They share the host kernel, so runtime controls work on syscalls, capabilities and kernel hooks, and an escape (ATT&CK T1611) lands on the node.
  • Prevent first, then detect. RuntimeDefault seccomp, an AppArmor or SELinux profile, dropped capabilities and a read-only root filesystem remove whole classes of attack before any sensor has to notice them.
  • eBPF is now the default sensor technology. Falco's modern eBPF probe is its default driver, and the legacy eBPF probe was deprecated in Falco 0.43.0; Tetragon filters and enforces inside the kernel.
  • Immutability is the best detection signal. A container should run only what was in its image, so a new executable, an interactive shell or a package manager at runtime is a high-fidelity alert.
  • Runtime alerts need Kubernetes context and an identity response. Pod, namespace, image and service account must travel with every alert, and containment includes revoking the credentials the workload could reach.

The runtime threat model

Runtime controls address what happens after a workload starts. The table maps the main container techniques in MITRE ATT&CK v20 to what a runtime layer can see or stop. Image and registry threats such as T1525 Implant Internal Image belong mainly to the Container Image Security Guide, and workload creation through the API to the Kubernetes Admission Control Guide.

ATT&CK techniqueWhat it looks like at runtimeRuntime signal or control
T1609 Container Administration Commandkubectl exec or docker exec into a running containerShell spawned with a terminal attached inside a container; Kubernetes audit log for the pods/exec subresource
T1059.004 Unix ShellA web process spawns sh or bashUnexpected parent and child process pair (D3FEND D3-PSA Process Spawn Analysis)
T1105 Ingress Tool Transfercurl or wget pulls a binary, which is then executedExecution of a file not present in the image layers; execution from /tmp or /dev/shm
T1611 Escape to HostAbuse of privileged mode, a mounted host path, the cgroup release_agent file or a kernel bugWrites to release_agent, mount syscalls, kernel module loading, setns; blocked by seccomp, capabilities and LSM profiles
T1552.007 Container APIReading the mounted service account token and calling the API serverOutbound connection from a pod to the Kubernetes API that the workload never makes in normal operation
T1552.001 Credentials In FilesSearching for cloud credential files and private keysReads of .aws/credentials, SSH keys or kubeconfig files by an unexpected process
T1613 Container and Resource DiscoveryEnumerating pods, nodes and secrets from inside a podAPI calls from a workload identity outside its normal pattern
T1496.001 Compute HijackingA cryptominer consuming CPUNew process with sustained CPU, connections to mining pools or the Stratum protocol
T1685 Disable or Modify Tools (tactic TA0112 Defense Impairment)Killing or unloading the security agent, clearing logsSensor heartbeat loss, DaemonSet pod deletion, log truncation events

The TeamTNT worm shows how short this chain can be: it started containers through exposed Docker APIs, installed XMRig, loaded a rootkit, cleaned logs and uploaded AWS credential files with curl. Every one of those steps is a runtime event. See the TeamTNT Worm 2020 analysis.

Kernel-level hardening: the preventive layer

Most runtime attacks rely on syscalls, capabilities or host access the workload never needed. Removing them is cheaper and more reliable than detecting their abuse. The Kubernetes Pod Security Standards encode the baseline; the controls below are the ones that matter most at runtime.

  • Seccomp: set securityContext.seccompProfile.type: RuntimeDefault on every pod. Without it, and without kubelet seccomp defaulting, Kubernetes runs containers Unconfined. The kubelet --seccomp-default flag (stable since Kubernetes v1.27) makes RuntimeDefault the node default. Default profiles differ between containerd and CRI-O and between releases, so test after runtime upgrades. Use Localhost profiles under the kubelet seccomp directory for tighter, workload-specific allowlists, and build them with a profile whose defaultAction is SCMP_ACT_LOG before switching to SCMP_ACT_ERRNO.
  • AppArmor or SELinux: since Kubernetes v1.30, AppArmor is set with the appArmorProfile field in the security context (earlier versions used annotations). The Baseline standard forbids Unconfined AppArmor and restricts SELinux types to container_t, container_init_t and container_kvm_t. These mandatory access controls limit file paths, mounts and raw network access even for a process running as root.
  • Capabilities: Restricted requires capabilities.drop: ["ALL"] and allows adding back only NET_BIND_SERVICE. CAP_SYS_ADMIN in particular enables many escape paths, including the cgroup release_agent technique.
  • Privilege and identity: privileged: false, allowPrivilegeEscalation: false, runAsNonRoot: true and a non-zero runAsUser. No hostPID, hostIPC, hostNetwork or hostPath volumes for application workloads.
  • Immutable filesystem: readOnlyRootFilesystem: true with explicit emptyDir mounts for scratch space. This turns "write a binary and run it" into a visible failure and makes drift detection precise.
  • Sandboxed runtimes: a RuntimeClass maps pods to a different runtime handler, such as a user-space kernel (gVisor) or a lightweight VM (Kata Containers). Use them for untrusted or multi-tenant code; RuntimeClass can declare pod overhead so scheduling accounts for the extra cost.

These settings are enforced at admission time through Pod Security Admission or a policy engine, which is why runtime and admission controls should be designed together. Seccomp and AppArmor appear in D3FEND as D3-SCF System Call Filtering and D3-EI Execution Isolation.

How eBPF runtime sensors work

eBPF lets small, verified programs run inside the Linux kernel, attached to syscalls, tracepoints and kernel functions. The verifier checks each program before it loads, so a bug in a sensor cannot crash the kernel the way a faulty kernel module can. Events are passed to user space through BPF maps and ring buffers, where a rule engine enriches them with container and Kubernetes metadata.

Collection methodHow it worksStrengthsLimits
Kernel moduleLoadable module capturing syscallsWorks on old kernels (Falco requires 3.10 or later)Needs full privileges, must match the kernel build, a fault can crash the node; often blocked on managed or locked-down nodes
Legacy eBPF probeeBPF program compiled per kernelSafer than a moduleNeeds kernel headers or prebuilt probes; deprecated in Falco 0.43.0 and may be removed in later releases
Modern eBPF probe (CO-RE)"Compile Once, Run Everywhere" program embedded in the agent, using BTF and the BPF ring bufferNo per-kernel build, can run with CAP_SYS_BPF, CAP_SYS_PERFMON, CAP_SYS_RESOURCE and CAP_SYS_PTRACE instead of full root on supporting kernelsNeeds a kernel with BTF and ring buffer support, usually 5.8 or later
In-kernel filtering and enforcement (Tetragon style)Policies attach to kprobes, fentry, tracepoints, uprobes or LSM BPF hooks and filter in the kernelLow overhead for high-frequency events; can act synchronouslyPolicies are closer to kernel internals, so authoring and testing take more skill
Sandbox event sourceEvents from a user-space kernel such as gVisorNo kernel access neededCoverage depends on the sandbox; Falco's gVisor engine was deprecated in 0.43.0
  • Falco: a rule engine over syscall and plugin events. With the modern eBPF probe (engine.kind: modern_ebpf, the default), ring buffer size is tuned with engine.modern_ebpf.buf_size_preset and engine.modern_ebpf.cpus_for_each_buffer. Falco detects and alerts; blocking is done by other tools reacting to its output.
  • Tetragon: a Cilium project that filters in the kernel and is Kubernetes-aware, so policies can be scoped to namespaces and pods. A TracingPolicy selector can run Sigkill, which terminates the calling process synchronously, or Override, which returns an error code from the hooked function (argError) instead of running it. Both execute inside the kernel BPF code.
  • Commercial runtime agents: most CNAPP and container security products now use eBPF sensors too. Ask which hooks they use, whether they enforce in the kernel or after the fact, and what they drop under load. See the CNAPP Guide.

Writing and tuning detection rules

A Falco rule has five required keys: rule, desc, condition, output and priority. Conditions are built from fields such as evt.type, proc.name, proc.pname, container.id and Kubernetes metadata, and reuse macro and list elements. Priorities run from EMERGENCY to DEBUG. Optional exceptions express tuned allowances as structured data rather than ever-longer conditions, and tags carry a maturity level and MITRE technique IDs.

Upstream Falco rule (stable set)What it catchesTuning note
Terminal shell in containerInteractive shell with a TTY, typically kubectl execAllow named break-glass users through exceptions; never disable outright in production namespaces
Drop and execute new binary in containerExecution of a file not in the base image (upper overlay layer)Only works where the runtime uses overlayfs; very few legitimate hits in immutable images
Detect release_agent File Container EscapesWrites to the cgroup release_agent file, a known escape pathTreat every hit as critical
Contact K8S API Server From ContainerA container talking to the API serverAllowlist operators and controllers that legitimately call the API
Read sensitive file untrustedReads of /etc/shadow and similar filesExclude known agents by image, not by process name alone
Fileless execution via memfd_create; Execution from /dev/shmIn-memory and tmpfs execution used by loaders and minersRare in normal workloads
Find AWS Credentials; Search Private Keys or PasswordsCredential hunting commandsPair with cloud audit log review of every key the pod could reach
Linux Kernel Module Injection Detected; PTRACE attached to processRootkit loading and process injectionShould be blocked by capabilities in containers; alert as critical
  • Start from behaviour, not signatures: the strongest rules encode invariants of your platform, such as "no package manager runs in production", "only the ingress controller listens on a new port" or "this namespace never contacts the API server".
  • Tune with exceptions keyed on image and namespace: process names are trivially spoofed (T1036 Masquerading); image digest, namespace and service account are not under the attacker's control in the same way.
  • Treat rules as code: keep custom rules in version control, review them like application changes and replay recorded events or test pods against them in CI before rollout.
  • Use profiling where available: learned per-workload baselines (expected processes, files and connections) catch drift that generic rules miss, but need a defined learning window and a way to accept legitimate change after deployments.

Detect only or enforce: choosing a response model

ApproachUse whenTrade-offs
Detect and alertNew deployments, mixed or legacy workloads, small teams building confidenceNo availability risk; attacker may finish an action before a human responds
Automated response outside the kernel (kill pod, cordon node, apply a deny NetworkPolicy, scale to zero)Stateless workloads with replicas; well-understood high-fidelity rulesSeconds of delay; response tooling needs its own privileged credentials, which must be protected
In-kernel enforcement (seccomp, LSM, Tetragon Sigkill or Override)Known-bad actions with no legitimate use: writes to release_agent, module loading, writing to sensitive host filesSynchronous and reliable, but a mistaken policy breaks production immediately
Sandboxed runtime (RuntimeClass)Untrusted code, CI runners, multi-tenant platforms, notebooksPerformance overhead and some syscall or device incompatibility

Small organisations usually get most value from strict pod security settings, a managed or open source sensor with the upstream stable rules, and alerts routed to the existing on-call channel. Large organisations should add custom rules per platform tier, automated containment for a short list of critical rules, sandboxed runtimes for untrusted workloads and a detection engineering process with test coverage for every rule. In both cases, prefer pod deletion over in-place repair: a compromised container is replaced from a known-good image, and its evidence is captured first.

Where identity fits

Almost every serious runtime incident turns into an identity incident within minutes, because the most valuable thing inside a container is usually a credential.

  • Service account tokens: a pod with an automatically mounted token lets an attacker call the API server as that workload. In the Kubeflow cryptomining attacks, the notebook's mounted service account was allowed to deploy containers, which is how the miner spread. Set automountServiceAccountToken: false where the API is not needed. See the Kubernetes Service Account Token Guide.
  • Cloud credentials: long-lived keys in environment variables or files are the first thing credential-hunting scripts look for. Use workload identity federation instead. See the Cloud Workload Identity Guide.
  • The sensor's own identity: runtime agents run as privileged DaemonSets with broad API read access. Protect their service accounts, restrict who can modify their configuration and alert when the DaemonSet is deleted or its pods stop reporting.
  • Enrich alerts with identity: include the pod's service account, its RBAC bindings and the cloud role it can assume, so responders know immediately what to revoke. See the Kubernetes RBAC Guide.

How to implement

  1. Inventory and baseline: list clusters, node kernel versions (BTF and ring buffer support decide which sensor driver works), container runtimes and namespaces. Run Pod Security Admission in audit and warn mode at the Restricted level to see which workloads need privileged settings.
  2. Harden by default: enable kubelet seccomp defaulting on a pilot node pool, then roll it out pool by pool. Fix workloads that fail rather than exempting them. Rollback is a kubelet configuration change on the affected pool.
  3. Deploy the sensor in detect-only mode: install it as a DaemonSet on a pilot pool with the upstream stable rules only, forward events to your SIEM with pod, namespace, image and service account fields, and set resource limits and priority class so the agent is not evicted first under pressure.
  4. Tune for two to four weeks: add exceptions keyed on image and namespace, write platform-invariant rules, and record every tuning decision in version control.
  5. Write response playbooks: for each critical rule define who is paged, what evidence is captured (process tree, container filesystem diff, network connections, the pod spec), how the pod is isolated with a deny NetworkPolicy, and which credentials are revoked.
  6. Introduce enforcement narrowly: add in-kernel blocking only for actions with no legitimate use, first in a staging cluster, then on one production pool with a documented rollback (remove the policy, which takes effect immediately).
  7. Test continuously: run scheduled, safe simulations (a shell in a test pod, a read of a canary credential file, a write to a test path) and alert if the expected detection does not fire.

Detection and measurement

  • Sensor coverage: percentage of nodes with a healthy runtime sensor, target 100% of production nodes, with alerts on any node that stops reporting.
  • Event loss: dropped-event counters from the sensor (ring buffer overflows). Persistent drops mean blind spots and need buffer or rule changes.
  • Hardening coverage: percentage of running pods with a non-Unconfined seccomp profile, all capabilities dropped and a read-only root filesystem.
  • Detection test pass rate: share of scheduled simulations that produced the expected alert within the target time.
  • Mean time to contain: from critical runtime alert to pod isolated and credentials revoked.
  • Alert precision: share of critical runtime alerts confirmed as true positives or authorised activity; low precision means rules need tuning before they are trusted for automation.

Correlate runtime events with the Kubernetes audit log (exec, pod creation, RBAC changes) and cloud audit logs for the workload's role. A shell in a container followed by API calls from the same service account and new cloud API activity is the pattern that turns three medium alerts into one incident.

Common failure patterns

  • Privileged pods treated as normal: monitoring agents, CI runners and storage drivers run privileged with host mounts, and so does a long tail of application pods that were "temporarily" exempted. Each one is a one-step escape route (T1611).
  • Sensor installed, rules never tuned: teams deploy the default ruleset, drown in noise from operators and agents, then mute whole rules instead of adding scoped exceptions, and the shell that matters goes unseen.
  • Exceptions keyed on process names: an allowlist for python or bash in a namespace is an allowlist for the attacker. Key exceptions on image digest, namespace and service account.
  • Mutable containers: images that install packages at start-up or write code to the root filesystem make "new binary executed" useless as a signal. The TeamTNT worm depended on installing and running new tools inside compromised hosts and containers.
  • No identity step in containment: responders delete the pod but leave the service account token, cloud keys and any secrets the pod read valid, so the attacker continues from outside the cluster.
  • Blind spots nobody measures: nodes with unsupported kernels, Windows nodes, serverless containers and event drops under load are silently uncovered unless coverage and drop metrics are tracked.

Framework mapping

FrameworkReferenceHow runtime security supports it
NIST CSF 2.0DE.CM-09, DE.AE-02, PR.PS-01, PR.PS-05Monitoring runtime environments; analysing adverse events; configuration baselines; preventing unauthorised software execution
NIST SP 800-53 Rev. 5SI-4, AU-12, CM-7, SC-39, SI-7, IR-4System monitoring; audit record generation; least functionality (seccomp, capabilities); process isolation; integrity (drift); incident handling
NIST SP 800-190Container runtime and host OS countermeasuresApplication Container Security Guide covering runtime, orchestrator and host risks
CIS Controls v8.1Control 4, Control 8, Control 13Secure configuration; audit log management; network monitoring and defence
ISO/IEC 27001:2022 Annex AA.8.9, A.8.16, A.8.19Configuration management; monitoring activities; installation of software on operational systems
MITRE D3FENDD3-SCF, D3-PSA, D3-EAL, D3-EISystem call filtering; process spawn analysis; executable allowlisting; execution isolation
MITRE ATT&CK v20T1609, T1610, T1611, T1552.007, T1496.001, T1685Container techniques the runtime layer detects or blocks

For the full lifecycle view of SP 800-190, see the NIST SP 800-190 Container Security Guide, and for cluster-wide controls the Kubernetes Security Guide.

Practitioner checklist

  • Enable kubelet seccomp defaulting (RuntimeDefault) on every node pool and forbid Unconfined seccomp and AppArmor at admission.
  • Drop all capabilities, block privilege escalation and run as non-root for application pods; keep a reviewed, expiring list of privileged exceptions.
  • Use read-only root filesystems so any new executable at runtime is an alert, not noise.
  • Deploy an eBPF runtime sensor on 100% of production nodes, preferring CO-RE drivers, and alert on sensor loss and event drops.
  • Enrich every runtime alert with pod, namespace, image digest, node and service account.
  • Keep custom rules and exceptions in version control, keyed on image and namespace, with tests in CI.
  • Restrict in-kernel enforcement to actions with no legitimate use, piloted in staging with a written rollback.
  • Include credential revocation (service account tokens, cloud roles, secrets read) in every container compromise playbook.
  • Run scheduled detection simulations and track the pass rate.
  • Use RuntimeClass sandboxes for untrusted, multi-tenant and notebook workloads.

Standards and references

Related NHI Mgmt Group resources: Container Security Guide · Linux Detection and Response Guide · Kubernetes Security Guide · CNAPP Guide · Top 10 Container Security Risks

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Written and reviewed by Lalit Choda, NHI Mgmt Group. Last updated 24 October 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.