Security teams should use runtime observability to baseline normal workload activity, then watch for abnormal network connections, unexpected file access, unusual process creation, and prohibited shell access. The value is not raw telemetry volume but correlated runtime context across pods and VMs. That context helps teams distinguish policy violations, compromise attempts, and misconfigurations after static checks have already been passed.
Why runtime observability beats static checks once workloads are live
Runtime observability is most useful when the control question changes from “was this workload configured correctly?” to “what is it actually doing now?” In Kubernetes and VM estates, static policies can approve a workload that still behaves badly at runtime, so teams need evidence of process, network, filesystem, and session activity to catch drift, abuse, and compromise after deployment.
The practical win is correlation. A single process event or outbound connection often looks harmless on its own, but when you can tie it to the pod, node, image, namespace, host, and time window, you can separate expected lifecycle activity from behaviour that deserves investigation. That is what turns raw telemetry into security signal.
- Baseline each workload’s normal process tree, egress destinations, and file paths before relying on alerts.
- Treat repeated deviations as a detection problem, not just an operations anomaly, because the same symptom can indicate misconfiguration or active abuse.
- Use the same runtime view across Kubernetes and VMs so analysts can compare patterns without changing their investigative model.
What abnormal workload behaviour usually looks like in practice
For security teams, abnormal behaviour is less about one universal indicator and more about a cluster of suspicious changes. Unexpected shell access, new child processes, unusual outbound network connections, access to sensitive files that the workload normally does not touch, and execution paths that do not match the approved job are all common signals. The key is whether the behaviour fits the workload’s intended purpose and operating pattern.
In Kubernetes, that often means watching for actions that cross container boundaries, escape the normal entrypoint, or use interactive commands where none should exist. In VMs, the same logic applies to service daemons, scheduled tasks, and application workers that suddenly behave like admin sessions or maintenance tooling. Runtime context matters because the same command can be routine in one workload and highly abnormal in another.
For container-specific depth, the NIST SP 800-190 Container Security guide remains a strong reference for runtime risk, while NHI visibility and over-privilege risks help explain why runtime monitoring often reveals issues that pre-deployment checks miss. The NHI management angle matters when workloads are using credentials, tokens, or service identities to perform actions on their own behalf.
How to turn runtime telemetry into usable detections
The most effective approach is to define behaviour in layers: workload identity, process behaviour, file activity, network behaviour, and privilege use. That lets teams write detections around meaningful deviation rather than around isolated events. For example, a pod that normally performs API calls and writes to a temp directory should not suddenly spawn a shell, read host files, and open an outbound session to an unfamiliar address.
Security teams should also tune for the environment they actually run. Kubernetes noise is different from VM noise, and a control that is too generic will drown analysts in false positives. Use image metadata, labels, deployment purpose, and time-based baselines to distinguish routine rollout activity from suspicious runtime changes. If your observability stack cannot tell a rollout from an intrusion, the problem is usually correlation quality rather than alert volume.
For teams building workload identity-aware telemetry, the SPIFFE workload identity specification is useful because it shows how strong workload identity and attestation can enrich runtime context. That context makes it easier to distinguish a legitimate service action from a process that is merely running inside the right cluster or VM.
Risk and Threat Considerations
Runtime observability is valuable because it surfaces post-deployment compromise paths that static review will not catch. If an attacker gains execution in a pod or VM, the first signs are often process spawning, outbound connectivity, credential use, or file access that deviates from the workload’s expected behaviour.
Failure mechanism: Teams rely on image scanning, admission policy, or baseline hardening alone, then miss the moment when a workload is abused at runtime through shell access, unexpected child processes, or suspicious egress.
Impact: A missed runtime anomaly can allow lateral movement, data access, secret abuse, or persistence, especially when the workload has broad network reach or inherited privilege.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-1 — Monitoring for Unusual Events | Runtime observability is used to detect abnormal workload behaviour in live systems. |
| Recommendation — Monitor workload runtime events for unusual process, network, and file activity. | ||
| CIS Controls v8 | 8.2 — Unusual Account Activity and Privilege Use Detection | Abnormal shell access and unexpected execution often reflect suspicious privilege use. |
| 10.2 — Automated Log Collection and Monitoring | Correlated runtime context depends on collecting and centralising workload telemetry. | |
| Recommendation — Detect and alert on abnormal runtime activity that indicates misuse of privileges or accounts. Centralise runtime logs and telemetry so pod and VM behaviour can be correlated quickly. | ||
| MITRE ATT&CK | T1059 — Command and Scripting Interpreter | Unexpected shell access and child processes are classic signs of runtime abuse. |
| T1049 — System Network Connections Discovery | Unusual outbound connections are a core runtime signal for compromise or misuse. | |
| T1005 — Data from Local System | Unexpected file access can indicate search, staging, or theft of local workload data. | |
| Recommendation — Hunt for interactive shell and scripting activity that should not occur in the workload. Investigate unexpected network connections from workloads that normally have narrow egress. Alert when workloads access files that fall outside their normal data-touch pattern. | ||
Practitioner Guidance
What to prioritise: Start with the workloads that have the widest blast radius, strongest external connectivity, or access to sensitive data and credentials. Those are the places where a small runtime deviation is most likely to matter operationally and security-wise.
What to verify: Confirm that analysts can answer three questions from one alert: what ran, where it ran, and what it touched. If the telemetry cannot connect process, network, and filesystem context, it is logging, not observability.
What good looks like: A mature program can suppress expected rollout activity, flag truly rare shell or file-access events, and explain whether the event is a policy violation, a misconfiguration, or active compromise without manual log stitching.
Practitioner takeaway: Runtime observability is most effective when it is built to explain workload intent, not just record events, because the security decision depends on deviation from expected behaviour in context.
Related resources from NHI Mgmt Group
- How should security teams implement runtime observability for AI agents in Kubernetes environments?
- How should security teams use workload inventory visibility to prioritise Kubernetes risk?
- How should security teams use eBPF to improve Kubernetes observability without adding too much overhead?
- How should security teams govern AI and workload identities at runtime?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 18, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
