Why do AI applications require traces, evals, and annotation instead of traditional metrics, logs, and traces alone?

Why This Matters for Security Teams

AI applications do not fail like conventional software, so traditional logs and infrastructure traces rarely explain whether the system was accurate, safe, or useful. Security and platform teams need traces, evals, and annotation because they create evidence about model inputs, outputs, tool calls, and human review decisions. That evidence supports governance, incident investigation, and controlled release decisions, especially when AI output affects customers, operations, or regulated workflows.

This is where observability becomes a trust problem as much as an engineering problem. A trace can show that an agent called the right API, but only an eval can show whether the result was materially correct, policy-compliant, or resilient to prompt injection. Annotation then closes the loop by turning edge cases into labelled examples that can improve future performance and reduce repeated failures. For control design, this aligns with the intent of NIST SP 800-53 Rev 5 Security and Privacy Controls, especially around auditability, accountability, and monitoring.

In practice, many security teams encounter AI failure only after a harmful output, a customer complaint, or an access misuse event has already occurred, rather than through intentional measurement.

How It Works in Practice

Traces, evals, and annotation serve different layers of assurance. Traces record the sequence of prompts, retrievals, tool executions, model responses, and guardrail decisions. Evals measure whether the result met a defined standard, such as factual accuracy, policy adherence, safe completion, or task success rate. Annotation adds contextual labels from reviewers, such as “hallucination,” “unsafe recommendation,” “needs escalation,” or “acceptable with caveat.” Together, they turn AI behaviour into something that can be tested, compared, and improved.

For security and operations teams, the practical workflow usually includes:

• Capturing request and response lineage for each significant AI interaction, including tool usage and retrieval context.
• Applying offline evals to known test sets before release and online evals to production traffic where appropriate.
• Separating functional quality metrics from security metrics, such as policy violations, sensitive data leakage, and jailbreak success.
• Using annotation to create reusable labels for failure analysis, incident triage, and model tuning.
• Tracking versioning for prompts, models, tools, datasets, and evaluation harnesses so results remain reproducible.

That approach is consistent with the governance emphasis in the NIST AI Risk Management Framework, which expects organisations to measure, manage, and document AI risk rather than rely on generic telemetry alone. It also reflects the attack-centric view in MITRE ATLAS, where adversarial behaviour against AI systems often appears in the path between input, model decision, and downstream action. For agentic systems, these controls overlap with tool-authorisation and action-review practices described in OWASP Agentic AI guidance.

These controls tend to break down when AI systems are loosely coupled to many external tools because the action chain becomes too long to reconstruct reliably after the fact.

Common Variations and Edge Cases

Tighter AI observability often increases storage, review, and privacy overhead, requiring organisations to balance assurance against operational cost. The right depth of tracing is not universal, and current guidance suggests that high-risk use cases need stronger lineage than low-risk internal assistants.

One common edge case is retrieval-augmented generation, where the model may appear correct because the answer is well written, but the real issue is whether the retrieval set was stale, biased, or incomplete. Another is autonomous agents, where a single user request can trigger many hidden steps. In that setting, a trace without evals can prove activity but not safety, and annotation without trace context can become subjective and inconsistent.

There is no universal standard for exactly which evals every AI system must run, but best practice is evolving toward risk-based coverage: factuality for knowledge tasks, refusal quality for unsafe requests, leakage detection for sensitive data, and tool-use validation for agentic workflows. For organisations in regulated environments, that usually means pairing engineering observability with governance review, retention rules, and change control. Where personal data or customer decisions are involved, the question is not just whether the model worked, but whether the evidence can support accountability after a dispute or incident.

For teams operating production AI at scale, the practical test is simple: if the system cannot be replayed, scored, and labelled, it cannot be improved with confidence.

Related resources from NHI Mgmt Group

• Why do clinical AI agents require runtime attestation instead of provisioning alone?
• Why do AI agents create a different access-risk profile than traditional applications?
• Why do AI agents create more leakage risk than traditional applications?
• Why do AI agents require stronger identity controls than standard applications?