OpenTelemetry will record that a model call happened, but it will not judge whether the output was faithful, safe, or policy-compliant. That means hallucinations, toxic content, PII exposure, and irrelevant answers can pass through as successful traces. Teams also miss enforcement because telemetry is passive and does not intercept or block responses.
Why This Matters for Security Teams
OpenTelemetry is excellent at showing that an AI request was made, how long it took, and where it travelled, but that is not the same as evaluating whether the response was acceptable. Without an ai evaluation layer, teams can mistake observability for assurance. A trace can confirm delivery while still allowing hallucinations, unsafe instructions, policy violations, or sensitive data leakage to reach the user. That gap matters because incident response, governance, and audit evidence all depend on knowing not just what happened, but whether the system behaved within approved boundaries.
The distinction is important in current guidance from the NIST Cybersecurity Framework 2.0, which treats visibility, control, and response as separate functions rather than interchangeable outcomes. OpenTelemetry supports the visibility side. It does not perform content inspection, policy scoring, or risk-based blocking. In practice, teams often discover the gap only after users have already consumed bad output, rather than through intentional pre-production evaluation.
How It Works in Practice
In an operational AI stack, OpenTelemetry should be treated as the evidence layer, not the decision layer. It captures spans, latency, token usage, dependencies, and service correlations. An AI evaluation layer sits beside or in front of that telemetry to assess the actual model output against quality, safety, and compliance criteria. For example, it can score factuality, detect prompt injection effects, flag PII, enforce tone or policy rules, and decide whether to allow, redact, retry, or escalate a response.
That separation is especially important for agentic systems, where a model may not only answer a question but also trigger tools, write code, or initiate actions. Telemetry may show tool invocation, but it will not determine whether the tool use was appropriate or authorised. Security teams usually need three functions working together: observability, evaluation, and enforcement.
- OpenTelemetry records the event chain and timing.
- AI evaluation tests the content and intent of the output.
- Policy enforcement blocks, rewrites, routes, or escalates risky responses.
To make that stack credible, organisations should align evaluation criteria to the same governance objectives described in the NIST AI Risk Management Framework, including validity, safety, accountability, and transparency. For adversarial techniques such as prompt injection, tool misuse, and output manipulation, the MITRE ATLAS knowledge base is useful for mapping attacks to detections and test cases. Where agentic workflows are involved, current practice is also moving toward agent-specific control sets such as OWASP Top 10 for LLM Applications, especially for prompt handling, excessive agency, and insecure output handling.
These controls tend to break down when high-volume, low-latency systems are allowed to ship telemetry only, because there is no inline decision point to stop unsafe output before delivery.
Common Variations and Edge Cases
Tighter AI evaluation often increases latency, engineering effort, and operational complexity, so organisations have to balance response speed against safety assurance. That tradeoff is real, especially when teams are trying to instrument many models or many prompts quickly. Best practice is evolving, but there is no universal standard for how much evaluation must happen synchronously versus asynchronously.
Some environments use lightweight checks only for low-risk content and reserve deeper evaluation for regulated workflows, customer-facing actions, or agentic tool use. Others run offline evaluation in CI or MLOps pipelines to catch regressions before deployment, then add runtime checks for the most sensitive routes. The right design depends on the impact of failure, the sensitivity of the data, and whether the model can take consequential actions.
OpenTelemetry alone is often enough for engineering troubleshooting, but not for governance over model behaviour. That matters most when the system handles personal data, generates customer communications, or acts as part of a larger workflow with permissions and side effects. In those cases, telemetry without evaluation can create a false sense of control, because everything appears measurable while the actual output quality remains unchecked.
Where identity and access are involved, teams should also treat model or agent credentials as governed assets, because the ability to call tools or retrieve data can become the real control boundary. For broader operational resilience, the same separation between logging and enforcement appears in frameworks such as ENISA publications, which consistently emphasise that monitoring must be paired with effective controls.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI governance requires evaluation, accountability, and ongoing measurement, not logging alone. | |
| MITRE ATLAS | Prompt injection and output manipulation need threat-informed testing beyond observability. | |
| OWASP Agentic AI Top 10 | Agentic workflows need controls for unsafe output, tool misuse, and excessive autonomy. | |
| NIST CSF 2.0 | DE.CM | Telemetry supports monitoring, but monitoring alone does not enforce safe behaviour. |
| NIST AI 600-1 | GenAI profiles emphasise testing and control of generative output quality and safety. |
Use AI RMF GOVERN and MAP activities to define evaluation gates before any model output reaches users.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 24, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
