MCP observability in Kubernetes still needs identity controls

At a glance

What this is: Stacklok's tutorial shows how MCP servers in Kubernetes can be instrumented with ToolHive, OTel, Prometheus, Grafana and Jaeger to expose metrics and traces that were previously hidden.

Why it matters: That matters because observability can reveal how MCP workloads behave, but IAM and NHI teams still need controls over session identity, tool invocation and privilege boundaries.

Context

MCP observability is the practice of turning tool calls, traces and runtime metrics into something operators can inspect, but visibility alone does not answer who is authorised to invoke a tool or whether session behaviour is safe. In Kubernetes, that gap matters because the control plane, the observability stack and the MCP proxy all sit close enough to become part of the trust boundary.

This article is about instrumenting MCP workloads so they stop behaving like black boxes, but the security question remains whether those workloads are governed as identities rather than just services. For identity teams, the relevant issue is not only what the server did, but what access path, session state and delegated authority allowed it to do it.

As MCP deployments spread, the pattern is becoming familiar: teams can measure activity long before they can govern it. That is typical of early-stage platform adoption, and it is exactly where identity controls usually lag behind operational visibility.

Key questions

Q: What breaks when MCP observability is missing?

A: Root cause analysis breaks first, followed by cost attribution, policy enforcement, and safe incident response. Teams lose the ability to distinguish model drift from tool failure, and they cannot tell whether sensitive data moved through a tool parameter even if the output looked harmless.

Q: Why do MCP servers need identity controls if telemetry already shows every request?

A: Because telemetry records behaviour after access has happened, while identity controls decide whether the caller should be trusted in the first place. In MCP, that distinction matters most for delegated sessions, where the same call pattern can be safe or unsafe depending on who initiated it and what permissions were attached.

Q: How can organisations tell whether MCP access is actually being governed?

A: A governed MCP deployment can answer who requested access, what scope was granted, when the token expires, and which tool calls were made under that token. If logs only show a shared credential or generic server activity, the organisation does not have effective identity governance for the protocol.

Q: What should teams do when Kubernetes-hosted MCP workloads become harder to audit?

A: Tighten the boundary around the proxy, make the caller identity explicit, and treat each tool path as a governed access route rather than a generic service endpoint. That combination reduces the chance that a well-instrumented cluster still hides overbroad delegated access.

Technical breakdown

How ToolHive turns MCP servers into observable services

ToolHive acts as a proxy layer in front of MCP servers, so it can collect telemetry without requiring every server implementation to expose native metrics and traces. In this architecture, the MCP server keeps handling tool requests, while ToolHive forwards operational data into an OpenTelemetry pipeline and standard backends such as Prometheus and Grafana. That separation is useful because it creates visibility at the boundary where requests enter the service. It also means the proxy becomes a control point for both observability and policy decisions. Practical implication: treat the proxy as part of the security architecture, not just as instrumentation.

Practical implication: place identity-aware controls at the MCP proxy boundary, not only in the application layer.

Why telemetry does not replace session identity and authorisation

Telemetry captures what happened after a request was accepted, but it does not itself prove that the caller should have been allowed to make that request. For MCP, the hard problem is not only logging tool calls, but binding those calls to a known session, a known client and a known set of permissions. That is why observability and authorisation are complementary rather than interchangeable. Kubernetes makes this easier to deploy, but not easier to trust, because the proxy, collector and backend can all faithfully record an unsafe access pattern. Practical implication: use telemetry to detect misuse, then enforce identity and entitlement checks before tool execution.

Practical implication: enforce client and session authorisation before MCP tool execution, then use telemetry for detection and audit.

MCP observability depends on the identity of the caller and the tool

MCP servers are increasingly acting like machine-facing services that accept delegated actions on behalf of a user or agent. That makes their identity model more important than a normal metrics dashboard would suggest. If the caller is not strongly identified, or if the tool is exposed without a clear access boundary, the resulting traces may be detailed but still insecure. In practice, observability can show usage patterns, yet it cannot distinguish legitimate delegation from overbroad access unless identity controls are built into the request path. Practical implication: pair MCP telemetry with workload identity and least-privilege access rules.

Practical implication: bind MCP access to workload identity and least privilege rather than relying on logs to reveal misuse.

Threat narrative

Attacker objective: The objective is to use a valid-looking MCP session to reach tools and workloads with more access than the caller should have had.

1. Entry occurs when a client or agent reaches the MCP proxy and establishes a session that can invoke tools inside the cluster.
2. Credential or session abuse would follow if that path is not tied to strong caller identity, because telemetry alone does not constrain what the session may do.
3. Impact is broadened observability without equivalent governance, which leaves tool execution, auditability and privilege boundaries easier to inspect than to trust.

Breaches seen in the wild

• Smithery.ai MCP hosting breach 2025: A Smithery.ai build flaw gave GitGuardian a live fly.io token controlling 3,000+ hosted MCP servers and the API keys their clients sent.
• Carbonato botnet 2026: A botnet puts a Hermes AI agent on exposed Docker hosts to steal AI API keys, SSH keys and tokens, powering the operators' own LLM gateway.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Observability without identity is only partial control: the article is correct that MCP servers become operationally understandable once ToolHive, OTel and standard dashboards are in place, but understandability is not governance. Telemetry can show tool calls, latency and trace paths, yet it cannot decide whether a caller should exist, whether a session should continue, or whether a delegated action is inside policy. For practitioners, the implication is that visibility programs must be designed alongside access governance from the start.

MCP introduces a session governance problem, not just a logging problem: the interesting security shift here is that MCP activity is session-shaped, which means the useful unit of control is often the invocation context rather than the server itself. That creates a stronger case for binding access to authenticated workload or agent identities, then auditing the session as a governed object. The field should stop treating proxy logs as the main control and start treating them as evidence after the control has already been enforced.

ToolHive-style proxy layers sharpen the need for boundary controls: when a proxy mediates all tool access, it becomes the right place to attach authorisation, observability and revocation logic together. If those functions are separated, teams get a false sense of safety from richer metrics while the actual decision point remains weak. The practical conclusion is to design the proxy as a policy enforcement point, not merely a telemetry relay.

MCP is revealing a broader machine identity problem in Kubernetes: these deployments are effectively machine-to-machine service paths that behave like delegated identities, even when teams describe them as simple application traffic. That makes NHI governance relevant now, not later, because the same issues seen in service accounts and API keys appear again in agent-mediated tool use. Practitioners should expect MCP to widen the identity surface unless they define ownership, scope and revocation for every tool path.

Ephemeral visibility can mask persistent privilege: the article’s stack can show every call, but a fully instrumented system can still preserve overly broad access for the whole lifetime of the session. That is the named gap here: telemetry-visible privilege, where the control is observable but not bounded. The practitioner takeaway is to govern privilege lifetime explicitly instead of assuming observability has reduced risk.

From our research library:

• 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, the protocol's first year of widespread adoption, according to the State of Secrets Sprawl 2026.
• Read next: NHI Authentication Guide

What this signals

Telemetry-visible privilege is now a real control gap: teams can often see MCP activity before they can prove that each tool call was properly authorised. That matters because observability programs tend to grow faster than identity policy, leaving a gap where access is measurable but not yet bounded.

MCP deployments also reinforce a familiar Kubernetes pattern: once a service becomes a proxy for delegated action, the operational question shifts from monitoring to ownership. Identity teams should expect pressure to define who owns MCP sessions, who can revoke them, and how those decisions are audited across namespaces and clusters.

For practitioners

• Define MCP session-level access policy Require a clear identity bound to every MCP client session and authorize tool use before the request reaches the server process. Separate telemetry from the allow decision so logs are evidence, not the control itself.
• Treat the MCP proxy as a policy enforcement point Place enforcement logic at the proxy boundary so tool invocation, revocation and audit all share the same control plane. This is where delegated access can be narrowed without rewriting each MCP server.
• Track session behaviour as an identity signal Use traces and metrics to flag unusual tool sequences, repeated invocations and unexpected client patterns, then map those signals back to the session identity that created them.
• Scope Kubernetes access around MCP workloads Limit which namespaces, service accounts and operators can deploy or modify MCP servers, because cluster-level permissions can expand the blast radius of a weak MCP access model.
• Review delegated tool access alongside rotation and revocation Make sure any credentials or tokens that support MCP access have a defined owner, expiry and revocation path so observability does not become a substitute for lifecycle control.

Key takeaways

• MCP observability improves runtime visibility, but it does not by itself establish who is authorised to use a tool or session.
• The real governance problem is binding delegated MCP activity to identity, scope and revocation so that access is controlled before execution.
• In Kubernetes, the proxy layer is the right place to combine enforcement with audit evidence, because telemetry alone cannot prevent overbroad access.

Key terms

• Mcp session: An MCP session is a live interaction between an AI agent and a tool or data source using the Model Context Protocol. It carries messages, requests, and responses for a bounded period, while preserving context, permissions, and state needed for the agent to act safely and consistently.
• Package Proxy: A package proxy is an intermediary service that handles software package requests between a sandbox and external registries. It can reduce direct internet exposure, but it also becomes a critical dependency if it is the only allowed egress path. If compromised, it can become the weak link in the containment chain.
• Delegated Access: Delegated access is permission granted to one identity to act on behalf of another user, service, or system. In NHI environments, this usually appears in OAuth-connected apps and automation tooling. It is powerful, but it must be tightly scoped and reviewed because it can persist long after the original business need ends.
• Workload Identity: The identity assigned to a software workload, such as a containerised application, serverless function, or microservice, enabling it to authenticate to other services without storing static credentials.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management. It is designed for practitioners who need to connect identity control to the broader security programme around Kubernetes and agentic systems.