Runtime observability for MCP servers exposes the security gap

At a glance

What this is: This guide argues that MCP security depends on runtime observability across protocol, OS execution, and agent behaviour, not on performance telemetry alone.

Why it matters: That matters to IAM practitioners because MCP tools can touch secrets, credentials, and privileged resources, which makes identity, access, and runtime control inseparable in agentic environments.

By the numbers:

• Only 18% of MCP server deployments implement any form of access scoping for tool permissions.
• 53% of MCP servers expose credentials through hard-coded values in configuration files.

👉 Read ARMO's guide to runtime observability for MCP servers

Context

MCP runtime observability is about seeing what a tool actually does after a request arrives, not just whether the request succeeded. In Model Context Protocol environments, the security gap appears when protocol logs, host execution, and agent decisions sit in separate monitoring systems that cannot explain whether a tool touched sensitive files, opened unexpected connections, or changed behaviour after a prompt.

That gap matters because MCP tools can act like privileged software interfaces into cloud and enterprise environments, which puts secrets, access scopes, and delegated trust under pressure. The article’s core point is that traditional observability answers performance questions while security needs evidence of actual runtime behaviour, a pattern that is increasingly familiar in AI-assisted operations and NHI governance. For practitioners, the starting assumption is typically incomplete visibility, not mature cross-layer detection.

Key questions

Q: What breaks when MCP observability is missing?

A: Root cause analysis breaks first, followed by cost attribution, policy enforcement, and safe incident response. Teams lose the ability to distinguish model drift from tool failure, and they cannot tell whether sensitive data moved through a tool parameter even if the output looked harmless.

Q: Why do MCP tools complicate least-privilege governance?

A: MCP tools break access into small capabilities, which is helpful, but only if each tool has a clear owner, scope, and approval model. Without that, organisations inherit a growing set of permissions that are difficult to inventory and even harder to revoke. Least privilege fails when the organisation cannot see the individual capability boundaries.

Q: How do security teams know if MCP runtime monitoring is actually working?

A: They should test whether one investigation can reconstruct the full chain from protocol request to host behaviour to agent decision. If the team still needs separate tools to explain file access, network egress, and tool sequencing, the observability model is incomplete. Effective monitoring produces one coherent attack story.

Q: What should teams do when an MCP tool starts behaving outside its baseline?

A: Contain the tool process, review the last correlated protocol and execution events, and compare the current tool-call sequence to baseline behaviour before restoring access. The key is to investigate the runtime path, not just the request log, because the malicious action may already have occurred inside the session.

Technical breakdown

Why protocol logs miss MCP security events

MCP protocol telemetry captures JSON-RPC requests, responses, method names, and parameters, but it stops at the interface boundary. That means a valid-looking request can still trigger sensitive file reads, database queries, or outbound transfers inside the tool process. From a security perspective, the protocol layer proves invocation, not intent or side effect. Correlation IDs, client identity, and capability state are essential because they preserve context when telemetry moves from the wire to execution and then to the agent. Without that bridge, defenders see activity, not evidence.

Practical implication: retain protocol metadata with stable correlation IDs so tool calls can be tied to host and agent telemetry.

How execution-layer telemetry reveals compromised tool behaviour

Execution-layer monitoring is the ground truth layer because it observes syscalls, process lineage, file access, and network egress regardless of what the application reports. In MCP environments, that matters when a tool is compromised or instrumented to hide its own behaviour. eBPF is useful here because it can collect kernel events without modifying the application code, which reduces the trust placed in the tool itself. When a tool reads credential files and then opens unusual outbound connections, the sequence becomes visible as a runtime attack story rather than a vague anomaly.

Practical implication: monitor file access and network egress at the kernel layer for every high-risk MCP tool process.

Why agent tool-call sequences are a security signal

Agent behaviour adds the missing decision layer. A single tool invocation may be normal, but an unusual chain of tools, retries, or scope changes can indicate prompt injection, session abuse, or capability drift. The article’s key insight is that no one layer is enough on its own: protocol anomalies can be noise, execution alerts can be context-free, and agent traces can look benign unless they are correlated with host activity. This is where cross-layer correlation turns separate observations into a usable detection narrative.

Practical implication: baseline normal tool-call graphs and alert when agent sequencing changes align with execution-layer anomalies.

Threat narrative

Attacker objective: The attacker wants to move from legitimate-looking tool invocation to covert secrets access and data exfiltration across the MCP runtime stack.

1. Entry occurs when an attacker supplies a prompt or session input that induces an MCP tool call with normal-looking JSON-RPC parameters.
2. Escalation happens when the invoked tool accesses sensitive files, broadens its scope, or opens unexpected network connections during execution.
3. Impact follows when correlated tool behaviour and agent chaining enable secrets exposure or data exfiltration without a clear signal in any single telemetry layer.

NHI Mgmt Group analysis

Runtime observability is now part of identity governance for MCP environments. MCP tools are not just application components. They are delegated actors that can touch files, credentials, and network destinations on behalf of an agent or user session. That makes runtime evidence relevant to IAM, PAM, and NHI governance because access decisions are only meaningful if defenders can see how the delegated tool actually behaved. Practitioners should treat observability as a control input, not just an operations function.

Cross-layer correlation is the named control gap this topic exposes: protocol certainty without execution proof is not security. The article shows why a valid MCP request, a normal latency spike, and a benign-looking tool trace can still conceal credential theft or exfiltration. This is a governance failure mode, not just a visibility issue, because it allows teams to mistake permitted invocation for safe behaviour. The practitioner conclusion is that security telemetry must survive the protocol-to-host boundary.

eBPF-based runtime visibility is becoming the practical baseline for proving what MCP tools really do. Application instrumentation can be stripped or falsified if the tool itself is part of the threat model. Kernel-level observation changes the trust model by moving evidence outside the process that may be compromised. For organisations building agentic AI controls, that shifts assurance from declared behaviour to observed behaviour, which is the only defensible standard in high-risk workflows.

AI agent tool chains create an NHI-like governance problem even when the tool is not formally autonomous. The article is a reminder that delegated software can accumulate effective privilege through sequences of valid actions, even if no single step looks dangerous. That is exactly where identity governance, access scoping, and runtime enforcement need to converge. Practitioners should assume that tool identity, tool privilege, and agent intent must be governed together, not separately.

What this signals

Runtime visibility will become a governance requirement, not just a detection preference. As MCP and agentic workflows spread, teams will need evidence that a delegated tool did only what the session allowed. The operational signal is whether your controls can join identity context, workload identity, and runtime telemetry before an incident forces that join. The article’s lesson is that observability without authority mapping will not satisfy security or audit teams.

Cross-layer monitoring creates a new control pattern for agentic systems: observe first, then constrain. That approach aligns with what the AI security ecosystem is converging on, including the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10. The practical signal is simple: if a tool can be invoked but its downstream behaviour cannot be explained, the control set is still immature.

Privilege scoping for tool identities is emerging as the hidden control plane in agent security. MCP servers, AI agents, and the credentials they use should be treated as governed identities with observable lifecycle and runtime limits. Where teams already manage service accounts and workload identities, those controls should extend into agent tooling so a valid call does not become a silent data path.

For practitioners

• Implement cross-layer correlation for MCP telemetry Join protocol logs, kernel execution events, and agent traces into a single incident timeline so a tool invocation can be traced from request to file access to outbound connection. Use correlation IDs that persist across all three layers.
• Instrument high-risk tools at the kernel layer Apply eBPF-based monitoring to tool processes that can read files, query data stores, or reach the network. Prioritise processes handling secrets, credential stores, and admin functions because application logs cannot prove what they touched.
• Baseline tool-call graphs by agent and workflow Record normal tool sequencing, retry patterns, and capability use for each agent or MCP workflow, then alert on drift when sequences jump from benign tools to file-read or network-send actions.
• Tie MCP visibility to IAM and workload identity data Map MCP clients, tool identities, namespaces, and session context back to workload identity and access records so security teams can answer who requested the action, which tool ran, and under what authority.

Key takeaways

• MCP security fails when teams rely on performance telemetry instead of runtime evidence of file access, network egress, and tool sequencing.
• Cross-layer correlation is the difference between noisy alerts and a credible attack story that security teams can investigate and defend.
• Identity, access, and runtime monitoring must converge for MCP tools because delegated software can exercise real privilege even when each individual action looks normal.

Key terms

• Model Context Protocol: Model Context Protocol is an open protocol that lets AI agents connect to tools and data sources. It expands what an agent can reach, so governance has to cover not only the model and its prompts, but also every system that can receive or return agent-driven data.
• Cross-Layer Correlation: The process of linking events from endpoint, SaaS, cloud, and identity systems into one coherent action path. It is not just log centralisation. Correlation turns separate observations into usable identity evidence for investigations, access review, and privilege governance.
• Runtime Observability Gap: The disconnect between what identity systems think was granted and what access systems show was actually used. This gap weakens governance because teams cannot confidently decide whether access is still necessary, especially in hybrid and distributed environments.
• eBPF: eBPF is a Linux kernel technology that lets security tools observe selected system events without modifying the kernel itself. In runtime protection, it is used to watch syscalls, process activity, and host behaviour, which helps detect post-exploitation actions and policy violations.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

• Kernel-level eBPF instrumentation patterns for Kubernetes-hosted MCP tools
• Example correlation logic that joins protocol events, execution telemetry, and agent traces
• Detection logic for prompt injection, capability drift, and abnormal tool chaining
• Operational retention guidance for execution-layer telemetry versus protocol logs

👉 ARMO's full post covers the MCP attack-story workflow, layer-by-layer signals, and implementation details.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It gives practitioners a framework for governing delegated access across identities, tools, and runtime workflows.