MCP observability in Kubernetes still needs identity controls

At a glance

What this is: This is a hands-on tutorial for deploying ToolHive with OTel, Prometheus, Grafana, and Jaeger to make MCP servers observable in Kubernetes.

Why it matters: It matters because visibility into MCP runtime behaviour is useful, but IAM and NHI teams still need explicit control over tool permissions, session scope, and audit trails.

👉 Read Stacklok's tutorial on deploying ToolHive with OTel, Prometheus, and Grafana

Context

MCP servers often behave like black boxes because they do not expose the same native metrics and traces that platform teams expect from standard services. That makes it harder to separate normal tool use from overreach, especially when MCP workloads are connected to sensitive data sources or privileged actions in Kubernetes.

The core governance issue is not just observability, but whether the organisation can explain what an MCP server did, which session did it, and whether that activity stayed inside the intended access boundary. Where MCP is used in identity-adjacent workflows, the control question quickly becomes one of NHI governance as much as platform monitoring.

Key questions

Q: How should teams govern MCP servers once they are exposed through Kubernetes CRDs?

A: Treat MCP servers like any other governed identity surface. Use reusable CRDs for shared OIDC, telemetry, and tool permissions, assign ownership to each server or catalog entry, and review who can add backends or widen discovery. The goal is to keep identity, observability, and authorization controls stable as the fleet grows.

Q: Why do MCP servers need more than standard observability?

A: Standard observability shows health, latency, and usage, but it does not prove whether a tool call was in scope or properly authorised. MCP environments need context about the session, caller, namespace, and tool permission model so teams can separate expected automation from privilege creep or misuse.

Q: What breaks when MCP traffic is visible but not tied to identity context?

A: You can see that something happened, but you cannot reliably say who or what initiated it, whether the request was allowed, or whether it exceeded its intended boundary. That weakens investigation, audit, and access review because the evidence is operational, not governable.

Q: How do security teams decide whether MCP observability is enough?

A: MCP observability is enough only when it produces actionable evidence for ownership, scope, and session-level behaviour. If the telemetry cannot show who invoked a tool, which client initiated the call, and whether the action matched the approved workflow, then the organisation still lacks governable identity evidence.

Technical breakdown

How ToolHive inserts observability in front of MCP servers

ToolHive acts as a proxy layer between MCP clients and servers, so requests and responses can be instrumented without depending on the server itself to emit rich telemetry. In this pattern, OpenTelemetry carries traces and metrics to a collector, while Prometheus scrapes the metrics output for dashboards and alerting. That gives operators a consistent view of request volume, latency, and service behaviour even when the underlying MCP server is a simple tool wrapper rather than a fully instrumented application.

Practical implication: place the observability layer where requests are brokered, not only where the workload runs.

Why the OpenTelemetry collector matters for MCP telemetry

The collector is the routing and enrichment point for telemetry before it reaches Prometheus or a trace backend. In this tutorial, it uses OTLP intake, kubeletstats for node and container signals, and Kubernetes attribute enrichment so telemetry can be tied back to pod and namespace context. That is important because raw metrics alone rarely explain whether abnormal tool activity came from a specific session, replica, or namespace boundary.

Practical implication: enrich telemetry with Kubernetes context early so investigations can map activity to workload identity and placement.

What Prometheus and Jaeger add to MCP operational visibility

Prometheus is used for metric storage and querying, which makes it suitable for trend analysis, service health, and alert rules. Jaeger handles traces, which helps operators inspect individual request paths and see how MCP interactions traverse the proxy and backend services. Together, they let teams compare steady-state behaviour with specific session activity, which is useful when a workload appears healthy but is still making unusual or overly broad tool calls.

Practical implication: combine metrics and traces so you can distinguish service health from suspicious usage patterns.

NHI Mgmt Group analysis

Observable is not the same as governed. ToolHive plus OTel improves visibility into MCP traffic, but visibility only tells you what happened after the fact. For identity teams, the missing question is whether the tool call was authorised at the right scope, for the right session, and with the right lifecycle controls. Observability helps with detection and forensics, but governance still depends on tool-scoped entitlement design and audit-ready session attribution.

Tool-brokered MCP architectures create a new control point for NHI oversight. When an MCP proxy sits in front of tool execution, it becomes the place where service identity, session identity, and access context converge. That makes the proxy a natural enforcement and evidence layer for NHI governance, especially where AI agents or automated clients are calling tools on behalf of users or workloads. The practical conclusion is that teams should treat the proxy as part of the identity plane, not just the networking stack.

Telemetry is becoming a prerequisite for AI agent accountability. The more agentic workflows depend on MCP, the more difficult it becomes to prove whether a tool call was benign automation or privilege overreach. A named concept here is MCP observability debt: the gap between having a functional tool chain and having enough session, action, and context data to govern it. Practitioners should close that gap before agent usage spreads across sensitive environments.

Kubernetes-native observability will increasingly shape MCP adoption patterns. Teams are unlikely to standardise MCP if they cannot plug it into existing platform controls, dashboards, and response workflows. That means observability features are now part of the governance conversation, because they determine whether security, platform, and identity teams can share a common operational record. Organisations that cannot correlate runtime events to access intent will struggle to scale MCP safely.

Identity and platform teams need a shared evidence model for MCP workloads. This tutorial shows how to collect signals, but the next governance step is deciding which events count as access evidence, which count as misuse, and which should trigger review. That is where NHI governance intersects with platform observability most directly. Teams should define the evidence standard before production rollout, not after incident response forces the issue.

What this signals

MCP observability debt: teams can operate tool-heavy AI systems without being able to govern them. The programme signal is clear: if your platform stack cannot bind session data, tool usage, and ownership into one record, you do not yet have enough evidence to support identity review or incident reconstruction.

As MCP spreads into production AI workflows, the governance boundary moves closer to the proxy, collector, and trace backend. That creates a practical opportunity for IAM and NHI teams to define where runtime evidence becomes access evidence, and to align that boundary with existing audit and review processes.

Security teams should also consider how MCP telemetry will feed broader standards-based monitoring, including OpenTelemetry for collection and queryable traces for investigation. The immediate task is not more dashboards, but a usable chain of custody for tool activity.

For practitioners

• Instrument MCP broker points for auditability Capture session IDs, tool names, namespace context, and proxy events at the point where MCP requests are brokered so later investigation can tie activity to a specific workload path.
• Treat telemetry enrichment as an access-control dependency Use Kubernetes attribute enrichment and service identity mapping so metrics and traces can be correlated to the pod, namespace, and calling session that generated them.
• Define alert thresholds for abnormal tool breadth Set alerts on unusual request volume, repeated high-risk tool calls, and sessions that access more tools or endpoints than the approved use case requires.
• Connect observability to NHI review workflows Feed proxy logs and trace evidence into entitlement review and offboarding processes so MCP access can be reviewed as a governed identity event, not just an application log line.

Key takeaways

• ToolHive plus OTel improves MCP visibility, but observability alone does not prove that tool access stayed within policy.
• The proxy layer becomes a natural governance point when MCP workloads need session attribution, audit trails, and identity context.
• Teams should connect telemetry to entitlement review and incident workflows before MCP usage expands into sensitive production environments.

Key terms

• Model Context Protocol: Model Context Protocol is an open protocol that lets AI agents connect to tools and data sources. It expands what an agent can reach, so governance has to cover not only the model and its prompts, but also every system that can receive or return agent-driven data.
• Tool Broker: A control layer that sits between a client and a tool-bearing service to mediate access, logging, and policy enforcement. For MCP environments, it is often the best place to capture session context, enforce scope, and create an audit trail for later review.
• Telemetry Enrichment: Telemetry enrichment is the process of adding context to raw security data so it is more useful for investigation and response. That can include asset, identity, geolocation, or threat intelligence context, but enrichment must be controlled so it does not distort the original evidence record.
• MCP Observability Debt: The gap between being able to run MCP workloads and being able to explain or govern their runtime behaviour with adequate evidence. It usually appears when teams have dashboards, but cannot tie activity back to identity, session scope, or approved tool access.

What's in the full article

Stacklok's full blog post covers the deployment details this post intentionally leaves at the architecture level:

• Step-by-step Helm commands for deploying kube-prometheus-stack, the OpenTelemetry collector, Jaeger, and the ToolHive operator in a Kubernetes cluster
• Exact example values files for OTel, Prometheus, and Jaeger that are tailored to the tutorial architecture
• Sample MCPServer telemetry configuration showing the endpoint, service name, metrics, tracing, and sampling settings used in the demo
• PromQL and Grafana walkthroughs for viewing ToolHive-specific metrics and traces in the dashboards

👉 Stacklok's full tutorial includes the exact Kubernetes manifests, telemetry settings, and dashboard steps

Deepen your knowledge

NHI Mgmt Group’s NHI Foundation Level course covers NHI governance, machine identity security, IAM, and secrets management in a practitioner-led format. It is designed for security and identity teams that need to connect access control, auditability, and lifecycle discipline across modern environments.