Teams miss behaviours they never described in advance, including stale retrieved context, repeated retries, and unresolved workflows. The agent can look healthy in dashboards while users experience wrong outcomes. Discovery has to start with trace clustering and manual confirmation, then move into scoring once the failure pattern is real and repeatable.
Why This Matters for Security Teams
Predefined scorers are useful for regression checks, but they only detect what the team already knows to measure. That creates a blind spot in agentic AI operations: the system may satisfy a narrow checklist while still failing in ways that affect users, workflows, or downstream controls. Current guidance from the NIST AI Risk Management Framework is clear that AI risk management has to account for context, performance, and impact, not just a static score.
In practice, this matters because agents fail across multiple layers at once. A tool call can succeed technically while the retrieved context is stale, the reasoning chain is off, or the workflow never completes. If monitoring only tracks predefined test cases, security and platform teams often miss the first sign of trouble until business users report broken outcomes or an incident review exposes the gap. That is especially dangerous where agents can take action, write back to systems, or trigger other automation. The OWASP Agentic AI Top 10 reflects this broader failure surface by treating agent-specific risks as operational and security issues, not just model-quality issues.
In practice, many security teams encounter agent failure only after a workflow has already been completed incorrectly, rather than through intentional detection.
How It Works in Practice
A scorer is usually a rule, model, or test harness that grades outputs against predefined expectations. That is helpful for known failure modes such as format drift, blocked content, or obvious tool misuse. It is not sufficient for agentic systems where the main risk is often emergent behaviour: repeated retries against the same tool, partial task completion, stale retrieval, silent delegation errors, or a loop that looks active but never converges.
Operationally, stronger monitoring starts with trace collection across prompts, tool calls, retrieval results, intermediate decisions, and final outputs. Teams then cluster traces to find repeated patterns, confirm whether a pattern represents true failure, and only then define a scorer or alert rule. This sequence matters because otherwise the scoring layer encodes assumptions too early and misses the unknown unknowns. That approach aligns with the threat-driven perspective in the MITRE ATLAS adversarial AI threat matrix and with agent-specific threat modeling in the CSA MAESTRO agentic AI threat modeling framework.
- Trace the full agent path, not just the final answer.
- Group repeated failures by workflow step, tool, or retrieval source.
- Use manual review to confirm whether the pattern is harmful, noisy, or expected.
- Promote confirmed patterns into scorers, alerts, and controls.
- Keep separate logic for quality issues, safety issues, and security issues.
This guidance tends to break down in highly dynamic environments where tool outputs, policies, or retrieval corpora change faster than the review process can confirm a stable failure pattern.
Common Variations and Edge Cases
Tighter scoring often increases monitoring overhead, requiring organisations to balance early detection against operational complexity. There is no universal standard for this yet, especially for multi-agent systems and long-running workflows where a single score can hide several distinct failure modes. Best practice is evolving toward layered evaluation rather than one score per agent.
Some environments need stronger emphasis on security rather than quality. For example, an agent that touches privileged systems should be assessed for harmful tool use, prompt injection susceptibility, and policy bypass, not only task completion. In those cases, the NIST AI Risk Management Framework and Anthropic — first AI-orchestrated cyber espionage campaign report both reinforce the need to think about misuse, not just correctness. The same logic applies when organisations map agent behaviours to control obligations under NIST SP 800-53 Rev 5 Security and Privacy Controls.
Where this gets tricky is that some failures are not repeatable enough to score immediately. A one-off retrieval miss, a transient tool outage, or a user-specific workflow exception may not justify a permanent control. In those cases, manual confirmation and trace review remain the right first step before hardening into automated detection.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | LLM07 | Predefined scorers miss agent-specific failure modes and misuse patterns. |
| NIST AI RMF | AI risk management requires context-aware evaluation beyond static metrics. | |
| MITRE ATLAS | ATLAS-TACTIC | Adversarial AI analysis helps model emergent failures and abuse paths. |
| NIST CSF 2.0 | DE.CM-1 | Continuous monitoring is needed when predefined tests miss operational failures. |
| NIST SP 800-53 Rev 5 | AU-6 | Audit review supports manual confirmation before automating detections. |
Use adversarial threat patterns to prioritise traces, detections, and abuse cases for agent review.
