Output-only evals score the final answer, not the work that produced it. An agent can return a polished summary while using stale data, repeating failed tool calls, skipping required steps, or claiming completion without verifying the underlying state change. Production reliability depends on the full trajectory, including retrieval, retries, permissions, tool behavior, and the final outcome.
Why the score can look fine while the agent is still unreliable
Output-only evaluation rewards the visible answer, so it can miss the hidden work that made the answer possible. That matters when the agent arrives at a plausible conclusion through brittle retrieval, repeated failed calls, stale context, or unverified assumptions. The system can appear correct even when the path to correctness is noisy, wasteful, or unsafe.
In practice, this creates a blind spot around trajectory quality. A single good-looking response does not tell you whether the agent respected permissions, used the right source at the right time, or recovered cleanly from tool failure. It also does not reveal whether the final answer is dependable across retries, different inputs, or changing system state.
What the hidden failure modes usually are
The most common failure modes are process failures, not just answer failures. An agent may cite stale data, loop on a broken tool call, skip a required verification step, or claim an action completed when the underlying system never changed. In agentic systems, that is often the difference between a harmless mistake and an incident.
These problems are especially easy to miss when the final response is polished. Natural language can conceal poor execution, so the evaluation needs to look at whether the agent actually retrieved, reasoned, acted, and confirmed the outcome. If the workflow depends on permissions, retries, or external state, the trajectory is part of the result, not background noise.
What good evaluation has to observe beyond the final answer
To assess reliability properly, you need signals that cover the full chain of behavior: retrieval quality, tool selection, retry handling, permission boundaries, state validation, and completion checks. A correct final sentence is only meaningful if the steps that produced it were also sound.
This is why trajectory-level evidence is more useful than answer-only grading for agent systems. It tells you whether the agent recovered from partial failure, whether it respected the limits of its access, and whether it confirmed the side effect it claimed to have caused. Without those observations, teams tend to overestimate reliability and underinvest in guardrails.
Risk and Threat Considerations
Output-only evals can create false confidence. That becomes risky when an agent has access to tools, secrets, operational actions, or downstream business workflows, because a convincing response can mask unauthorized actions, skipped checks, or failed state changes.
Failure mechanism: The evaluation optimizes for the surface artifact, so repeated tool errors, stale retrieval, permission failures, and unverified state transitions never influence the score. An attacker or bug can exploit that gap by producing a plausible final answer while the agent behaves unsafely underneath.
Impact: Teams may ship agents that look accurate in testing but fail under production conditions, where bad retries, hidden dependency failures, or unconfirmed actions can cause data loss, access misuse, or operational drift.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agent reliability failures often hide privilege misuse or unverified actions. |
| ASI02 — Tool Misuse | Output-only evals miss failed, repeated, or inappropriate tool behavior in the trajectory. | |
| ASI08 — Cascading Failures | A polished final answer can mask upstream failures that propagate through agent workflows. | |
| Recommendation — Inspect agent traces for privilege misuse and require outcome verification before trusting completion. Evaluate tool calls and retries, not just the final response, to catch misuse and brittle execution. Check for upstream failures and propagation paths when scoring agent reliability. | ||
| NIST AI RMF | Govern | Agent reliability needs governance over evaluation scope, monitoring, and accountability. |
| Recommendation — Define evaluation criteria that include trajectory evidence and operational accountability. | ||
| NIST SP 800-53 Rev 5 | AU-2 — Event Logging | Trajectory evaluation depends on logs of tool use, retries, and outcome evidence. |
| AU-6 — Audit Record Review, Analysis, and Reporting | Reviewing execution traces is necessary to detect hidden reliability failures. | |
| IA-9 — Service Authentication | Many agent failures are only material when tool and service access are part of execution. | |
| Recommendation — Log agent steps and outcomes so reliability can be reviewed beyond the final answer. Review execution records for repeated failures, stale inputs, and unconfirmed actions. Authenticate service-to-service actions so agent steps are attributable and bounded. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Reliable agent evaluation needs preserved logs of tool calls and state changes. |
| Recommendation — Centralize and retain agent execution logs for post-run reliability analysis. | ||
Practitioner Guidance
What to verify: Treat the final answer as one signal, not the control objective. Verify that the agent can show the retrieval source, tool trace, retry behavior, and the evidence that the claimed state change actually occurred.
What good looks like: A reliable agent leaves a defensible trail from prompt to outcome. If the answer is correct but the trace shows repeated failures, missing confirmations, or unnecessary privilege use, the system is not production-ready.
Decision rule: If the task can affect real data, accounts, or operational state, evaluate the trajectory first and the prose second. Output-only scoring is acceptable for low-stakes summarization, but it is too weak for agents that act.
Practitioner takeaway: The question is not whether the agent can sometimes produce the right answer, it is whether it can do so with controlled, repeatable, and verified execution.
