Notifications
Clear all
Topic starter
20/08/2026 5:30 pm
TL;DR: MCP servers can be turned from black boxes into observable Kubernetes services by pairing OpenTelemetry, Prometheus, Grafana, and Jaeger, according to Stacklok, but the operational win is only half the story because telemetry does not replace governance over tool access and session behaviour. The wider implication is that as MCP deployments scale, identity teams need to treat observability, privilege, and auditability as linked controls rather than separate projects.
NHIMG editorial — based on content published by Stacklok: Blog Integrations From black box to observable: Deploy ToolHive with OTel + Prometheus
Questions worth separating out
Q: How should teams govern MCP servers once they are exposed through Kubernetes CRDs?
A: Treat MCP servers like any other governed identity surface.
Q: Why do MCP servers need more than standard observability?
A: Standard observability shows health, latency, and usage, but it does not prove whether a tool call was in scope or properly authorised.
Q: What breaks when MCP traffic is visible but not tied to identity context?
A: You can see that something happened, but you cannot reliably say who or what initiated it, whether the request was allowed, or whether it exceeded its intended boundary.
Practitioner guidance
- Instrument MCP broker points for auditability Capture session IDs, tool names, namespace context, and proxy events at the point where MCP requests are brokered so later investigation can tie activity to a specific workload path.
- Treat telemetry enrichment as an access-control dependency Use Kubernetes attribute enrichment and service identity mapping so metrics and traces can be correlated to the pod, namespace, and calling session that generated them.
- Define alert thresholds for abnormal tool breadth Set alerts on unusual request volume, repeated high-risk tool calls, and sessions that access more tools or endpoints than the approved use case requires.
What's in the full article
Stacklok's full blog post covers the deployment details this post intentionally leaves at the architecture level:
- Step-by-step Helm commands for deploying kube-prometheus-stack, the OpenTelemetry collector, Jaeger, and the ToolHive operator in a Kubernetes cluster
- Exact example values files for OTel, Prometheus, and Jaeger that are tailored to the tutorial architecture
- Sample MCPServer telemetry configuration showing the endpoint, service name, metrics, tracing, and sampling settings used in the demo
- PromQL and Grafana walkthroughs for viewing ToolHive-specific metrics and traces in the dashboards
👉 Read Stacklok's tutorial on deploying ToolHive with OTel, Prometheus, and Grafana →
MCP observability in Kubernetes: what identity teams should watch?
Explore further
20/08/2026 6:18 pm
Observable is not the same as governed. ToolHive plus OTel improves visibility into MCP traffic, but visibility only tells you what happened after the fact. For identity teams, the missing question is whether the tool call was authorised at the right scope, for the right session, and with the right lifecycle controls. Observability helps with detection and forensics, but governance still depends on tool-scoped entitlement design and audit-ready session attribution.
A question worth separating out:
Q: How do security teams decide whether MCP observability is enough?
A: MCP observability is enough only when it produces actionable evidence for ownership, scope, and session-level behaviour. If the telemetry cannot show who invoked a tool, which client initiated the call, and whether the action matched the approved workflow, then the organisation still lacks governable identity evidence.
👉 Read our full editorial: MCP observability in Kubernetes still needs identity controls
