MCP servers expose the next observability gap in Kubernetes

At a glance

What this is: This is a Stacklok analysis of why MCP servers create a new observability blind spot in Kubernetes, with the key finding that many servers provide little or no native telemetry.

Why it matters: It matters because identity, access, and runtime governance for AI agents and non-human identities depends on being able to see which services acted, when, and through which tools.

By the numbers:

• Only 18% of MCP server deployments implement any form of access scoping for tool permissions.
• 53% of MCP servers expose credentials through hard-coded values in configuration files.
• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job.
• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems.

👉 Read Stacklok's analysis of the MCP observability gap in Kubernetes

Context

MCP server observability is now part of the broader governance problem in Kubernetes because runtime visibility, access scoping, and accountability all depend on telemetry. When a service can handle requests, manage state, and mediate tools without exposing metrics or structured logs, platform teams lose the evidence needed to separate benign automation from risky behaviour.

The article frames a real operational gap: traditional Prometheus and OpenTelemetry patterns cover many workloads, but they do not solve the case where an MCP server has no native telemetry surface at all. That gap intersects directly with NHI and agentic AI governance because tool-bearing services can behave like non-human identities even when they look like ordinary application components. For teams already managing workload identity, this is a familiar but more opaque version of the same control problem.

Stacklok's starting point is typical of the problem space, not an outlier: the issue is not that observability tools failed, but that the protocol and implementation pattern often leave nothing meaningful to observe.

Key questions

Q: What breaks when MCP servers do not expose telemetry in Kubernetes?

A: Teams lose the ability to trace tool use, identify which service handled a request, and distinguish normal automation from risky behaviour. That undermines incident response, performance analysis, and governance at the same time. In practice, an unobservable MCP server becomes a black box that hides both operational faults and privilege misuse.

Q: Why do MCP servers create governance problems for AI workloads?

A: Because they mediate tool access for AI systems while often leaving little trace of what happened inside the interaction. That makes it harder to prove safe use, investigate failures, or confirm that permissions stayed within intended bounds. For security teams, the lack of evidence is itself a control weakness.

Q: How do security teams decide whether MCP observability is enough?

A: MCP observability is enough only when it produces actionable evidence for ownership, scope, and session-level behaviour. If the telemetry cannot show who invoked a tool, which client initiated the call, and whether the action matched the approved workflow, then the organisation still lacks governable identity evidence.

Q: Who should be accountable for MCP server permissions and agent actions?

A: Accountability should sit with the business or platform owner that approved the data path, plus the security team that set the guardrails. If the agent can access regulated data or take sensitive actions, the organisation needs a clear owner for the server, the credentials, and the downstream consequences of misuse.

Technical breakdown

Why MCP servers become black boxes in Kubernetes

Kubernetes observability assumes workloads expose something measurable, such as metrics, logs, or traces. MCP servers often break that assumption because they are designed as lightweight brokers for context and tool access, not as fully instrumented application tiers. If the server does not publish a /metrics endpoint, structured logs, or trace headers, Prometheus and OpenTelemetry can only see surrounding infrastructure, not the server's own decision path. In practice, that means the control plane can show pod health while hiding tool selection, request routing, and state changes.

Practical implication: require telemetry hooks at build time for every MCP server, not after deployment.

How Prometheus and OpenTelemetry help, and where they stop

Prometheus is strong at scraping numeric time series from known endpoints, while OpenTelemetry can unify metrics, logs, and traces across services. Together they give teams both aggregate service health and request-level diagnostics. Their limitation is simple: they still depend on the workload exposing instrumented signals. When an MCP server omits those signals, the observability stack can correlate everything around the service but not the service itself, which creates a blind spot in incident response and change analysis.

Practical implication: treat missing telemetry as a deployment defect, not an acceptable implementation choice.

Why MCP observability is also an identity governance issue

MCP servers sit at the boundary between AI systems and the tools they can invoke, which makes them part of the identity and privilege path even when they are not named that way. A server that can handle tool requests without scoped permissions, auditability, or traceable context effectively becomes an ungoverned non-human control point. That is why observability and identity governance now converge in agentic AI environments: if you cannot observe the tool broker, you cannot reliably govern the brokered access.

Practical implication: map MCP servers into NHI and agentic AI governance reviews alongside service accounts and workload identities.

Threat narrative

Attacker objective: The objective is to operate through an invisible control point so tool use, privilege abuse, or malicious automation can proceed without reliable detection or attribution.

1. Entry occurs when an AI system reaches an MCP server that is exposed inside the Kubernetes environment but lacks meaningful telemetry or access scoping.
2. Escalation happens when the server mediates tool access without clear permission boundaries, making it difficult to distinguish approved actions from overreach.
3. Impact follows when teams cannot trace which server, tool, or upstream request produced unexpected behaviour, delaying containment and root-cause analysis.

NHI Mgmt Group analysis

Observability gaps are now governance gaps when AI systems act through MCP servers. The article shows that telemetry is not just an operations concern. When tool mediation happens inside infrastructure that does not expose metrics, logs, or traces, security teams lose the evidence needed to govern non-human behaviour. That makes MCP visibility a control-plane requirement for NHI and agentic AI programmes, not a nice-to-have for SRE teams.

MCP servers create a named concept we should treat seriously: the tool-broker black box. These services sit between AI systems and downstream tools, but their minimal interfaces often hide the very events practitioners need to audit. The implication is straightforward. If the broker cannot be observed, it cannot be confidently authorised, reviewed, or investigated.

Prometheus and OpenTelemetry remain necessary but insufficient controls for AI-era Kubernetes. They can show that the platform is healthy while leaving the most privileged request path invisible. That means teams need to extend governance to the application layer, especially where MCP servers introduce tool access, state handling, and implied delegation. Practitioners should treat telemetry gaps as an identity and access risk, not merely an observability limitation.

Workload identity discipline becomes more important as MCP adoption grows. Even when an MCP server is not an NHI in the strict sense, it behaves like one in the access path because it brokers tool use on behalf of another system. That makes workload identity, audit logging, and scoped permissions part of the same control story. Teams that separate observability from identity governance will miss the real failure mode.

The market signal is clear: agentic AI infrastructure is forcing a convergence between runtime visibility and access governance. The next category of control will not be only about discovering workloads or tracing requests. It will also be about proving who or what had authority to invoke a tool, when that authority was exercised, and whether the system left a usable audit trail. Practitioners should prepare for that convergence now.

What this signals

Tool-broker black box: MCP visibility gaps will push security teams to treat observability as an authorisation control, not just an operations layer. That shift matters because the same service that hides request paths can also hide privilege misuse, especially when AI systems are given broader access than human operators would receive.

The practical response is to align monitoring, identity review, and workload governance around the same runtime events. Where teams already use workload identity patterns, they should extend those controls to AI tool brokers and insist on traceable delegation paths.

As agentic AI adoption grows, the organisations that can prove who invoked what, through which broker, will be the ones able to investigate and contain failures quickly. The rest will only know something happened after the blast radius is already visible.

For practitioners

• Instrument MCP servers before production rollout Require each MCP server to expose metrics, structured logs, and trace context as part of the deployment gate. If a server cannot be instrumented, treat it as an exception that needs compensating controls, not a standard deployment.
• Map MCP tool access to identity governance controls Classify MCP servers as privileged broker points in your NHI and agentic AI inventory, then assign owners, approval paths, and audit requirements to each tool permission set.
• Tie observability alerts to access review triggers When telemetry shows unusual tool invocation patterns, route the event into access review and incident triage workflows so teams can determine whether the behaviour reflects misconfiguration, misuse, or compromise.
• Add telemetry requirements to procurement and engineering standards Make native observability a release criterion for any platform that handles AI tool use, including requirements for request correlation, log retention, and per-tool accountability.

Key takeaways

• MCP servers create a visibility gap because many do not expose the telemetry that Kubernetes observability assumes exists.
• The problem is also an identity issue because tool brokers mediate privileged access on behalf of AI systems.
• Teams need to make telemetry, auditability, and scoped access part of the same control design for agentic workloads.

Key terms

• Model Context Protocol: Model Context Protocol is an open protocol that lets AI agents connect to tools and data sources. It expands what an agent can reach, so governance has to cover not only the model and its prompts, but also every system that can receive or return agent-driven data.
• Runtime Observability Gap: The disconnect between what identity systems think was granted and what access systems show was actually used. This gap weakens governance because teams cannot confidently decide whether access is still necessary, especially in hybrid and distributed environments.
• Tool broker: A tool broker is any service that mediates access between an AI system and external tools, APIs, or stateful actions. Because it sits in the middle of delegated execution, it should be governed like a privileged control point, with logging, scoped permissions, and clear ownership.
• Workload Identity: The identity assigned to a software workload — such as a containerised application, serverless function, or microservice — enabling it to authenticate to other services without storing static credentials.

What's in the full article

Stacklok's full blog post covers the operational detail this post intentionally leaves for the source:

• Prometheus scrape and service-discovery patterns for Kubernetes workloads that emit metrics
• OpenTelemetry Collector deployment models for correlating metrics, logs, and traces across clusters
• Practical examples of how MCP servers remain invisible when they do not expose structured telemetry
• The planned ToolHive approach for surfacing MCP tool-usage data in existing monitoring stacks

👉 Stacklok's full post covers Prometheus, OpenTelemetry, and the MCP visibility problem in more implementation detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners building controlled access paths. It helps identity and security teams connect runtime visibility to enforceable access policy.