TL;DR: MCP servers often behave as black boxes because they expose little or no telemetry, leaving operations teams unable to measure request rates, latency, errors, or tool usage, according to Stacklok. The article argues that proxy-based observability can close that gap without server changes, and that the real issue is governance of opaque AI-to-tool interfaces, not just dashboard coverage.
At a glance
What this is: Stacklok’s analysis shows that many MCP servers lack native telemetry, creating an observability gap that leaves AI tool traffic hard to measure and govern.
Why it matters: This matters because IAM, NHI and platform teams need evidence of what AI-connected services are doing, not just functional access, if they are going to govern risk and reliability.
Context
MCP servers are software services that expose tools and resources to AI systems, but many do so without the metrics and traces that modern operations teams expect. When a service can process requests but not report what it is doing, basic governance questions about volume, latency, failure rates and tool usage remain unanswered.
The identity angle is real because MCP sits at the boundary between AI systems and the services they call. That makes the telemetry gap more than an observability problem: it weakens the ability to govern AI-to-tool access, understand runtime behavior, and distinguish normal usage from risky or unexpected interactions.
Key questions
Q: What breaks when MCP servers do not expose telemetry in Kubernetes?
A: Teams lose the ability to trace tool use, identify which service handled a request, and distinguish normal automation from risky behaviour. That undermines incident response, performance analysis, and governance at the same time. In practice, an unobservable MCP server becomes a black box that hides both operational faults and privilege misuse.
Q: Why does missing observability increase risk for AI toolchains?
A: Because AI-connected services can trigger real actions in downstream systems, and those actions need attribution, measurement, and review. When telemetry is missing, teams cannot tell whether usage is routine, excessive, or unexpected, so policy enforcement becomes guesswork instead of evidence-based control.
Q: How should teams detect that MCP visibility is failing?
A: Look for absent metrics, missing traces, flat request counts, unexplained latency blind spots, and tool usage that cannot be tied back to specific workloads. Those signals indicate that the observability model is not seeing the protocol path clearly enough to support operations or governance.
Q: When should organisations use proxy-based observability for MCP services?
A: Use it when the server cannot expose telemetry itself, when production rollout cannot wait for code changes, or when the organisation needs standard metrics in an existing OTel and Prometheus pipeline. The proxy is most useful when observability must be added without disrupting the service.
Technical breakdown
Why MCP servers become black boxes in production
Model Context Protocol servers are designed to expose capabilities to AI clients, not necessarily to expose operational telemetry. In practice, many do not publish a /metrics endpoint, do not emit traces, and do not integrate cleanly with existing OTel or Prometheus pipelines. That leaves operators able to see traffic entering and leaving, but not the internal request path, tool selection, latency distribution, or error patterns. The result is a control gap between functional service delivery and observable service behavior, which matters when the server is part of a production AI workflow.
Practical implication: Treat lack of native telemetry as an architectural risk, not a dashboard inconvenience.
How proxy-based observability changes the control point
A proxy can observe requests and responses at the orchestration layer without requiring the underlying MCP server to change. That shifts measurement from application instrumentation to traffic interception, allowing request counts, duration, errors, and tool usage to be collected even when the server itself is opaque. In Kubernetes, that matters because the deployment layer already has service discovery, scaling, and routing context that can enrich telemetry. The trade-off is that the proxy becomes part of the trust boundary, so its configuration, resilience, and placement become governance concerns as well as technical ones.
Practical implication: Place observability controls where the traffic actually flows, then govern the proxy as part of the runtime boundary.
Why OTel and Prometheus are necessary but not sufficient
OpenTelemetry and Prometheus are strong operational standards, but they only work when systems emit data that can be collected. For MCP workloads, the problem is upstream of the monitoring stack: if the server does not surface metrics or traces, the stack has nothing to scrape or receive. A proxy that normalises MCP-specific events into standard telemetry formats solves that ingestion problem, but it does not remove the need for policy on what gets measured, retained, and alerted on. Observability here is also an accountability layer for AI-to-tool interaction.
Practical implication: Use standard telemetry pipelines, but extend governance to the collection layer and the semantics of MCP usage.
Threat narrative
Attacker objective: The practical objective is to operate or exploit AI-connected services without drawing enough operational attention to abnormal usage, failures, or tool invocation patterns.
- Entry occurs when AI clients send requests to MCP servers that expose little or no telemetry, so activity enters the environment without adequate operational visibility.
- Credential or privilege abuse is not the central issue here, but opaque tool calls can hide excessive or unexpected usage patterns that would otherwise trigger review.
- Impact is degraded detection, slower troubleshooting, and weaker governance over AI-to-tool interactions because teams cannot measure what the service is doing in production.
Breaches seen in the wild
- Smithery.ai MCP hosting breach 2025: A Smithery.ai build flaw gave GitGuardian a live fly.io token controlling 3,000+ hosted MCP servers and the API keys their clients sent.
- Secrets in Docker Hub images (RWTH Aachen study): A 2023 RWTH Aachen study found secrets in 8.5% of container images, and 275,269 internet hosts still using the leaked private keys.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Observability debt is becoming governance debt for AI-to-tool interfaces. When an MCP server cannot expose reliable telemetry, the organisation cannot prove how that server behaves under load, how often tools are invoked, or whether usage aligns with intended purpose. That makes the issue relevant to runtime governance, not just SRE hygiene. For identity and access programmes, the lesson is that trust in AI-connected services now depends on measurable behavior.
The control point has shifted from server internals to the interception layer. If the application will not instrument itself, a proxy becomes the only practical place to capture usage evidence and enforce visibility. That is a useful pattern, but it also means the observability layer becomes part of the security architecture and must be treated with the same care as any other access boundary. Practitioners should think in terms of governed mediation, not passive monitoring.
AI toolchains need a named concept: protocol-opaque service exposure. This is the condition where a service can accept and process protocol requests while remaining effectively unreadable to standard telemetry. In that state, operational teams lose the evidence needed for incident triage, anomaly detection, and policy enforcement. The practitioner conclusion is simple: if the protocol is visible to the model but invisible to operations, governance is incomplete.
Telemetry is now part of non-human identity governance. MCP servers and related AI service components behave like non-human actors in production, even when they are not identities in the narrow IAM sense. Their actions create access, tool use, and downstream effects that must be attributable. That means identity programmes need to care not only about issuance and authentication, but also about runtime observability of the entities that consume and broker access.
Standardisation only works when the upstream data is trustworthy. OTel and Prometheus can normalise collection and reporting, but they cannot manufacture signal that never existed. The field should stop treating observability as a backend concern and start treating it as a property of AI service design. Practitioners should require telemetry readiness before MCP services move into production.
From our research library:
- 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, the protocol's first year of widespread adoption, according to the State of Secrets Sprawl 2026.
- Read next: MCP Security Guide
What this signals
Protocol-opaque service exposure: MCP creates a class of production service that can be functionally available while remaining operationally unreadable, which forces governance to move closer to the traffic path. In practice, that means observability design has to be treated as part of AI service architecture, not as a post-deployment add-on.
The telemetry gap also changes how identity teams should think about non-human controls. If an MCP service can invoke tools but cannot be measured, reviewed, or attributed, the organisation is left with access that is technically active but operationally ungoverned.
Stacklok’s analysis aligns with a broader pattern in AI infrastructure: 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, the protocol's first year of widespread adoption, according to the State of Secrets Sprawl 2026. When the protocol layer and the telemetry layer are both weak, governance becomes reactive rather than preventive.
For practitioners
- Define telemetry readiness for MCP services Require every MCP server to declare how request counts, latency, tool usage, and error states will be observed before production approval.
- Insert a proxy at the runtime boundary Place an observability proxy where MCP traffic passes so that measurement does not depend on server-side code changes or instrumented endpoints.
- Correlate tool usage with workload identity Tie MCP request metrics to namespace, pod, and service context so teams can distinguish normal tool invocation from unexpected AI-driven behavior.
- Alert on missing or flat telemetry Treat absent traces, sudden drops in request counts, and unexplained error spikes as operational signals that the MCP control plane is blind.
- Review proxy trust and failure handling Validate how the proxy behaves if Prometheus, tracing backends, or downstream exporters are unavailable, because observability loss can become operational loss.
Key takeaways
- MCP servers that do not emit metrics or traces create a governance problem as much as an operational one, because teams cannot see how AI-to-tool interactions behave in production.
- Proxy-based observability can restore measurement without changing server code, but it also shifts trust and failure handling into the mediation layer.
- For identity and platform teams, the practical requirement is telemetry readiness before production, with attribution, alerting, and runtime accountability built into the MCP path.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | MCP configuration and runtime opacity often coexist with exposed secrets and missing telemetry. |
| NHI-10 — Human Use of NHI | MCP servers mediate human and AI access to tools, so their activity needs governed attribution. | |
| Recommendation — Scan MCP deployment artifacts for secret leakage and require telemetry before production rollout. Attribute MCP tool use to the workload or operator that initiated it and review abnormal access paths. | ||
| MITRE ATT&CK | TA0007;TA0010 — Discovery; Exfiltration | Opaque server behavior hinders discovery of abnormal tool activity and downstream data movement. |
| Recommendation — Map missing telemetry to discovery and exfiltration blind spots in detection engineering. | ||
| NIST CSF 2.0 | DE.CM-01 — Networks and network services are monitored to detect potential cybersecurity events | This article centers on monitoring gaps that prevent detection of abnormal service behavior. |
| Recommendation — Extend continuous monitoring to MCP traffic and treat missing metrics as a control failure. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | MCP observability depends on usable logs, traces, and usage records for investigation. |
| Recommendation — Standardise logging and trace collection for MCP services so investigations have durable evidence. | ||
Key terms
- Model Context Protocol: Model Context Protocol is an open protocol that lets AI agents connect to tools and data sources. It expands what an agent can reach, so governance has to cover not only the model and its prompts, but also every system that can receive or return agent-driven data.
- Protocol-opaque Service Exposure: Protocol-opaque service exposure is the condition where a service is reachable and functional but does not provide enough telemetry for operators to understand its behavior. The service may answer requests normally while remaining unreadable to standard monitoring, tracing, and governance controls.
- Gateway-Based Observability: Gateway-based observability is an approach where a gateway layer becomes the main collection point for telemetry across services. Instead of instrumenting every service independently for each monitoring or logging tool, teams place collection at the connectivity layer to simplify operations and reduce repetitive configuration.
- Telemetry Readiness: Telemetry readiness is the degree to which a service can expose the metrics, traces, and logs needed for reliable production operation. It is not just instrumentation; it is the ability to support troubleshooting, capacity planning, detection, and accountability before incidents force those questions.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the operational realities of AI-enabled services and service-to-service access.
Published by the NHIMG editorial team on August 20, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org