Image scanning and admission policy decide what is allowed to start, but nothing in that pipeline tells you what a container does once it is running. An attacker who lands in a pod through a vulnerable web framework, an exposed dashboard or a stolen service account token works inside a process that passed every pre-deployment check. Runtime security closes that gap: it restricts what a running container can do at the kernel level and watches what it actually does, so that a shell, a dropped binary or an escape attempt is blocked or seen within seconds. This guide covers the runtime layer for containers and Kubernetes: kernel hardening with seccomp, AppArmor and SELinux, sandboxed runtimes, eBPF sensors such as Falco and Tetragon, detection rules, response and measurement. It is written for platform engineers, detection engineers and SOC teams who run containers in production. Host Linux detection (auditd, EDR on the node) is covered in the Linux Detection and Response Guide, and the wider container lifecycle in the Container Security Guide.
Key takeaways
- Containers are processes, not boundaries. They share the host kernel, so runtime controls work on syscalls, capabilities and kernel hooks, and an escape (ATT&CK T1611) lands on the node.
- Prevent first, then detect. RuntimeDefault seccomp, an AppArmor or SELinux profile, dropped capabilities and a read-only root filesystem remove whole classes of attack before any sensor has to notice them.
- eBPF is now the default sensor technology. Falco's modern eBPF probe is its default driver, and the legacy eBPF probe was deprecated in Falco 0.43.0; Tetragon filters and enforces inside the kernel.
- Immutability is the best detection signal. A container should run only what was in its image, so a new executable, an interactive shell or a package manager at runtime is a high-fidelity alert.
- Runtime alerts need Kubernetes context and an identity response. Pod, namespace, image and service account must travel with every alert, and containment includes revoking the credentials the workload could reach.
The runtime threat model
Runtime controls address what happens after a workload starts. The table maps the main container techniques in MITRE ATT&CK v20 to what a runtime layer can see or stop. Image and registry threats such as T1525 Implant Internal Image belong mainly to the Container Image Security Guide, and workload creation through the API to the Kubernetes Admission Control Guide.
| ATT&CK technique | What it looks like at runtime | Runtime signal or control |
|---|---|---|
| T1609 Container Administration Command | kubectl exec or docker exec into a running container | Shell spawned with a terminal attached inside a container; Kubernetes audit log for the pods/exec subresource |
| T1059.004 Unix Shell | A web process spawns sh or bash | Unexpected parent and child process pair (D3FEND D3-PSA Process Spawn Analysis) |
| T1105 Ingress Tool Transfer | curl or wget pulls a binary, which is then executed | Execution of a file not present in the image layers; execution from /tmp or /dev/shm |
| T1611 Escape to Host | Abuse of privileged mode, a mounted host path, the cgroup release_agent file or a kernel bug | Writes to release_agent, mount syscalls, kernel module loading, setns; blocked by seccomp, capabilities and LSM profiles |
| T1552.007 Container API | Reading the mounted service account token and calling the API server | Outbound connection from a pod to the Kubernetes API that the workload never makes in normal operation |
| T1552.001 Credentials In Files | Searching for cloud credential files and private keys | Reads of .aws/credentials, SSH keys or kubeconfig files by an unexpected process |
| T1613 Container and Resource Discovery | Enumerating pods, nodes and secrets from inside a pod | API calls from a workload identity outside its normal pattern |
| T1496.001 Compute Hijacking | A cryptominer consuming CPU | New process with sustained CPU, connections to mining pools or the Stratum protocol |
| T1685 Disable or Modify Tools (tactic TA0112 Defense Impairment) | Killing or unloading the security agent, clearing logs | Sensor heartbeat loss, DaemonSet pod deletion, log truncation events |
The TeamTNT worm shows how short this chain can be: it started containers through exposed Docker APIs, installed XMRig, loaded a rootkit, cleaned logs and uploaded AWS credential files with curl. Every one of those steps is a runtime event. See the TeamTNT Worm 2020 analysis.
Kernel-level hardening: the preventive layer
Most runtime attacks rely on syscalls, capabilities or host access the workload never needed. Removing them is cheaper and more reliable than detecting their abuse. The Kubernetes Pod Security Standards encode the baseline; the controls below are the ones that matter most at runtime.
- Seccomp: set
securityContext.seccompProfile.type: RuntimeDefaulton every pod. Without it, and without kubelet seccomp defaulting, Kubernetes runs containersUnconfined. The kubelet--seccomp-defaultflag (stable since Kubernetes v1.27) makes RuntimeDefault the node default. Default profiles differ between containerd and CRI-O and between releases, so test after runtime upgrades. UseLocalhostprofiles under the kubelet seccomp directory for tighter, workload-specific allowlists, and build them with a profile whosedefaultActionisSCMP_ACT_LOGbefore switching toSCMP_ACT_ERRNO. - AppArmor or SELinux: since Kubernetes v1.30, AppArmor is set with the
appArmorProfilefield in the security context (earlier versions used annotations). The Baseline standard forbidsUnconfinedAppArmor and restricts SELinux types tocontainer_t,container_init_tandcontainer_kvm_t. These mandatory access controls limit file paths, mounts and raw network access even for a process running as root. - Capabilities: Restricted requires
capabilities.drop: ["ALL"]and allows adding back onlyNET_BIND_SERVICE.CAP_SYS_ADMINin particular enables many escape paths, including the cgrouprelease_agenttechnique. - Privilege and identity:
privileged: false,allowPrivilegeEscalation: false,runAsNonRoot: trueand a non-zerorunAsUser. NohostPID,hostIPC,hostNetworkorhostPathvolumes for application workloads. - Immutable filesystem:
readOnlyRootFilesystem: truewith explicitemptyDirmounts for scratch space. This turns "write a binary and run it" into a visible failure and makes drift detection precise. - Sandboxed runtimes: a RuntimeClass maps pods to a different runtime
handler, such as a user-space kernel (gVisor) or a lightweight VM (Kata Containers). Use them for untrusted or multi-tenant code; RuntimeClass can declare podoverheadso scheduling accounts for the extra cost.
These settings are enforced at admission time through Pod Security Admission or a policy engine, which is why runtime and admission controls should be designed together. Seccomp and AppArmor appear in D3FEND as D3-SCF System Call Filtering and D3-EI Execution Isolation.
How eBPF runtime sensors work
eBPF lets small, verified programs run inside the Linux kernel, attached to syscalls, tracepoints and kernel functions. The verifier checks each program before it loads, so a bug in a sensor cannot crash the kernel the way a faulty kernel module can. Events are passed to user space through BPF maps and ring buffers, where a rule engine enriches them with container and Kubernetes metadata.
| Collection method | How it works | Strengths | Limits |
|---|---|---|---|
| Kernel module | Loadable module capturing syscalls | Works on old kernels (Falco requires 3.10 or later) | Needs full privileges, must match the kernel build, a fault can crash the node; often blocked on managed or locked-down nodes |
| Legacy eBPF probe | eBPF program compiled per kernel | Safer than a module | Needs kernel headers or prebuilt probes; deprecated in Falco 0.43.0 and may be removed in later releases |
| Modern eBPF probe (CO-RE) | "Compile Once, Run Everywhere" program embedded in the agent, using BTF and the BPF ring buffer | No per-kernel build, can run with CAP_SYS_BPF, CAP_SYS_PERFMON, CAP_SYS_RESOURCE and CAP_SYS_PTRACE instead of full root on supporting kernels | Needs a kernel with BTF and ring buffer support, usually 5.8 or later |
| In-kernel filtering and enforcement (Tetragon style) | Policies attach to kprobes, fentry, tracepoints, uprobes or LSM BPF hooks and filter in the kernel | Low overhead for high-frequency events; can act synchronously | Policies are closer to kernel internals, so authoring and testing take more skill |
| Sandbox event source | Events from a user-space kernel such as gVisor | No kernel access needed | Coverage depends on the sandbox; Falco's gVisor engine was deprecated in 0.43.0 |
- Falco: a rule engine over syscall and plugin events. With the modern eBPF probe (
engine.kind: modern_ebpf, the default), ring buffer size is tuned withengine.modern_ebpf.buf_size_presetandengine.modern_ebpf.cpus_for_each_buffer. Falco detects and alerts; blocking is done by other tools reacting to its output. - Tetragon: a Cilium project that filters in the kernel and is Kubernetes-aware, so policies can be scoped to namespaces and pods. A
TracingPolicyselector can runSigkill, which terminates the calling process synchronously, orOverride, which returns an error code from the hooked function (argError) instead of running it. Both execute inside the kernel BPF code. - Commercial runtime agents: most CNAPP and container security products now use eBPF sensors too. Ask which hooks they use, whether they enforce in the kernel or after the fact, and what they drop under load. See the CNAPP Guide.
Writing and tuning detection rules
A Falco rule has five required keys: rule, desc, condition, output and priority. Conditions are built from fields such as evt.type, proc.name, proc.pname, container.id and Kubernetes metadata, and reuse macro and list elements. Priorities run from EMERGENCY to DEBUG. Optional exceptions express tuned allowances as structured data rather than ever-longer conditions, and tags carry a maturity level and MITRE technique IDs.
| Upstream Falco rule (stable set) | What it catches | Tuning note |
|---|---|---|
| Terminal shell in container | Interactive shell with a TTY, typically kubectl exec | Allow named break-glass users through exceptions; never disable outright in production namespaces |
| Drop and execute new binary in container | Execution of a file not in the base image (upper overlay layer) | Only works where the runtime uses overlayfs; very few legitimate hits in immutable images |
| Detect release_agent File Container Escapes | Writes to the cgroup release_agent file, a known escape path | Treat every hit as critical |
| Contact K8S API Server From Container | A container talking to the API server | Allowlist operators and controllers that legitimately call the API |
| Read sensitive file untrusted | Reads of /etc/shadow and similar files | Exclude known agents by image, not by process name alone |
| Fileless execution via memfd_create; Execution from /dev/shm | In-memory and tmpfs execution used by loaders and miners | Rare in normal workloads |
| Find AWS Credentials; Search Private Keys or Passwords | Credential hunting commands | Pair with cloud audit log review of every key the pod could reach |
| Linux Kernel Module Injection Detected; PTRACE attached to process | Rootkit loading and process injection | Should be blocked by capabilities in containers; alert as critical |
- Start from behaviour, not signatures: the strongest rules encode invariants of your platform, such as "no package manager runs in production", "only the ingress controller listens on a new port" or "this namespace never contacts the API server".
- Tune with exceptions keyed on image and namespace: process names are trivially spoofed (T1036 Masquerading); image digest, namespace and service account are not under the attacker's control in the same way.
- Treat rules as code: keep custom rules in version control, review them like application changes and replay recorded events or test pods against them in CI before rollout.
- Use profiling where available: learned per-workload baselines (expected processes, files and connections) catch drift that generic rules miss, but need a defined learning window and a way to accept legitimate change after deployments.
Detect only or enforce: choosing a response model
| Approach | Use when | Trade-offs |
|---|---|---|
| Detect and alert | New deployments, mixed or legacy workloads, small teams building confidence | No availability risk; attacker may finish an action before a human responds |
| Automated response outside the kernel (kill pod, cordon node, apply a deny NetworkPolicy, scale to zero) | Stateless workloads with replicas; well-understood high-fidelity rules | Seconds of delay; response tooling needs its own privileged credentials, which must be protected |
| In-kernel enforcement (seccomp, LSM, Tetragon Sigkill or Override) | Known-bad actions with no legitimate use: writes to release_agent, module loading, writing to sensitive host files | Synchronous and reliable, but a mistaken policy breaks production immediately |
| Sandboxed runtime (RuntimeClass) | Untrusted code, CI runners, multi-tenant platforms, notebooks | Performance overhead and some syscall or device incompatibility |
Small organisations usually get most value from strict pod security settings, a managed or open source sensor with the upstream stable rules, and alerts routed to the existing on-call channel. Large organisations should add custom rules per platform tier, automated containment for a short list of critical rules, sandboxed runtimes for untrusted workloads and a detection engineering process with test coverage for every rule. In both cases, prefer pod deletion over in-place repair: a compromised container is replaced from a known-good image, and its evidence is captured first.
Where identity fits
Almost every serious runtime incident turns into an identity incident within minutes, because the most valuable thing inside a container is usually a credential.
- Service account tokens: a pod with an automatically mounted token lets an attacker call the API server as that workload. In the Kubeflow cryptomining attacks, the notebook's mounted service account was allowed to deploy containers, which is how the miner spread. Set
automountServiceAccountToken: falsewhere the API is not needed. See the Kubernetes Service Account Token Guide. - Cloud credentials: long-lived keys in environment variables or files are the first thing credential-hunting scripts look for. Use workload identity federation instead. See the Cloud Workload Identity Guide.
- The sensor's own identity: runtime agents run as privileged DaemonSets with broad API read access. Protect their service accounts, restrict who can modify their configuration and alert when the DaemonSet is deleted or its pods stop reporting.
- Enrich alerts with identity: include the pod's service account, its RBAC bindings and the cloud role it can assume, so responders know immediately what to revoke. See the Kubernetes RBAC Guide.
How to implement
- Inventory and baseline: list clusters, node kernel versions (BTF and ring buffer support decide which sensor driver works), container runtimes and namespaces. Run Pod Security Admission in
auditandwarnmode at the Restricted level to see which workloads need privileged settings. - Harden by default: enable kubelet seccomp defaulting on a pilot node pool, then roll it out pool by pool. Fix workloads that fail rather than exempting them. Rollback is a kubelet configuration change on the affected pool.
- Deploy the sensor in detect-only mode: install it as a DaemonSet on a pilot pool with the upstream stable rules only, forward events to your SIEM with pod, namespace, image and service account fields, and set resource limits and priority class so the agent is not evicted first under pressure.
- Tune for two to four weeks: add exceptions keyed on image and namespace, write platform-invariant rules, and record every tuning decision in version control.
- Write response playbooks: for each critical rule define who is paged, what evidence is captured (process tree, container filesystem diff, network connections, the pod spec), how the pod is isolated with a deny NetworkPolicy, and which credentials are revoked.
- Introduce enforcement narrowly: add in-kernel blocking only for actions with no legitimate use, first in a staging cluster, then on one production pool with a documented rollback (remove the policy, which takes effect immediately).
- Test continuously: run scheduled, safe simulations (a shell in a test pod, a read of a canary credential file, a write to a test path) and alert if the expected detection does not fire.
Detection and measurement
- Sensor coverage: percentage of nodes with a healthy runtime sensor, target 100% of production nodes, with alerts on any node that stops reporting.
- Event loss: dropped-event counters from the sensor (ring buffer overflows). Persistent drops mean blind spots and need buffer or rule changes.
- Hardening coverage: percentage of running pods with a non-Unconfined seccomp profile, all capabilities dropped and a read-only root filesystem.
- Detection test pass rate: share of scheduled simulations that produced the expected alert within the target time.
- Mean time to contain: from critical runtime alert to pod isolated and credentials revoked.
- Alert precision: share of critical runtime alerts confirmed as true positives or authorised activity; low precision means rules need tuning before they are trusted for automation.
Correlate runtime events with the Kubernetes audit log (exec, pod creation, RBAC changes) and cloud audit logs for the workload's role. A shell in a container followed by API calls from the same service account and new cloud API activity is the pattern that turns three medium alerts into one incident.
Common failure patterns
- Privileged pods treated as normal: monitoring agents, CI runners and storage drivers run privileged with host mounts, and so does a long tail of application pods that were "temporarily" exempted. Each one is a one-step escape route (T1611).
- Sensor installed, rules never tuned: teams deploy the default ruleset, drown in noise from operators and agents, then mute whole rules instead of adding scoped exceptions, and the shell that matters goes unseen.
- Exceptions keyed on process names: an allowlist for
pythonorbashin a namespace is an allowlist for the attacker. Key exceptions on image digest, namespace and service account. - Mutable containers: images that install packages at start-up or write code to the root filesystem make "new binary executed" useless as a signal. The TeamTNT worm depended on installing and running new tools inside compromised hosts and containers.
- No identity step in containment: responders delete the pod but leave the service account token, cloud keys and any secrets the pod read valid, so the attacker continues from outside the cluster.
- Blind spots nobody measures: nodes with unsupported kernels, Windows nodes, serverless containers and event drops under load are silently uncovered unless coverage and drop metrics are tracked.
Framework mapping
| Framework | Reference | How runtime security supports it |
|---|---|---|
| NIST CSF 2.0 | DE.CM-09, DE.AE-02, PR.PS-01, PR.PS-05 | Monitoring runtime environments; analysing adverse events; configuration baselines; preventing unauthorised software execution |
| NIST SP 800-53 Rev. 5 | SI-4, AU-12, CM-7, SC-39, SI-7, IR-4 | System monitoring; audit record generation; least functionality (seccomp, capabilities); process isolation; integrity (drift); incident handling |
| NIST SP 800-190 | Container runtime and host OS countermeasures | Application Container Security Guide covering runtime, orchestrator and host risks |
| CIS Controls v8.1 | Control 4, Control 8, Control 13 | Secure configuration; audit log management; network monitoring and defence |
| ISO/IEC 27001:2022 Annex A | A.8.9, A.8.16, A.8.19 | Configuration management; monitoring activities; installation of software on operational systems |
| MITRE D3FEND | D3-SCF, D3-PSA, D3-EAL, D3-EI | System call filtering; process spawn analysis; executable allowlisting; execution isolation |
| MITRE ATT&CK v20 | T1609, T1610, T1611, T1552.007, T1496.001, T1685 | Container techniques the runtime layer detects or blocks |
For the full lifecycle view of SP 800-190, see the NIST SP 800-190 Container Security Guide, and for cluster-wide controls the Kubernetes Security Guide.
Practitioner checklist
- Enable kubelet seccomp defaulting (
RuntimeDefault) on every node pool and forbidUnconfinedseccomp and AppArmor at admission. - Drop all capabilities, block privilege escalation and run as non-root for application pods; keep a reviewed, expiring list of privileged exceptions.
- Use read-only root filesystems so any new executable at runtime is an alert, not noise.
- Deploy an eBPF runtime sensor on 100% of production nodes, preferring CO-RE drivers, and alert on sensor loss and event drops.
- Enrich every runtime alert with pod, namespace, image digest, node and service account.
- Keep custom rules and exceptions in version control, keyed on image and namespace, with tests in CI.
- Restrict in-kernel enforcement to actions with no legitimate use, piloted in staging with a written rollback.
- Include credential revocation (service account tokens, cloud roles, secrets read) in every container compromise playbook.
- Run scheduled detection simulations and track the pass rate.
- Use RuntimeClass sandboxes for untrusted, multi-tenant and notebook workloads.
Standards and references
- NIST: SP 800-190 Application Container Security Guide
- MITRE ATT&CK: Containers Matrix
- Kubernetes: Restrict a Container's Syscalls with seccomp
- Kubernetes: Pod Security Standards
- Falco: Kernel Events and Drivers
- Falco: Stable Default Rules
- Tetragon: Overview
- MITRE D3FEND: System Call Filtering
Related NHI Mgmt Group resources: Container Security Guide · Linux Detection and Response Guide · Kubernetes Security Guide · CNAPP Guide · Top 10 Container Security Risks