Final-response scoring can miss silent failures in the middle of the run. An agent may report success after a failed call, retry the same action, or leave behind duplicate records. Trajectory scoring and state assertions catch those problems by checking the recorded decisions, tool calls, errors, and resulting system state, not just the last message.
Why This Matters for Security Teams
When an MCP eval only scores the final response, it rewards the appearance of success instead of the actual security and operational path taken to get there. That is a serious problem for agentic systems because a tool call can fail quietly, a retry can duplicate state, or an unsafe action can happen even if the last message looks clean. The result is blind spots in incident review, compliance evidence, and production change control.
For teams evaluating OWASP Agentic AI Top 10, the issue is not just model quality. It is whether the evaluation can detect harmful trajectories, hidden exceptions, and side effects that a final answer cannot reveal. NHI Management Group’s research on OWASP Agentic Applications Top 10 treats this as a core control gap, because agent behaviour must be assessed across the whole run, not only at the end.
In practice, many security teams discover duplicate records, leaked secrets, or unauthorized tool use only after downstream systems have already absorbed the damage.
How It Works in Practice
Trajectory scoring evaluates the full chain of agent decisions: prompts, tool selections, tool outputs, intermediate errors, retries, and the final state. State assertions add a second layer by checking whether the environment ended in the expected condition, such as whether a record was created once, permissions stayed unchanged, or a file was not exposed. Final-response scoring cannot see these outcomes unless the last message explicitly reveals them, and agents often do not.
This is especially important in mcp environment because the protocol is about structured tool access, not just conversational output. If the agent calls the wrong tool, retries after a timeout, or writes to the wrong resource, the final answer may still read as successful. The evaluation needs to inspect the execution trace and compare it to intended policy.
- Score the full trajectory, not only the final natural-language response.
- Assert expected post-conditions in the target system after each task.
- Check tool call order, argument values, and retry behaviour for anomalies.
- Flag hidden failures, duplicates, and unauthorized side effects as distinct outcomes.
These controls align with the risk patterns described in OWASP Top 10 for Agentic Applications 2026, especially where tool misuse and unsafe execution paths matter more than the final answer. The broader operational context is also covered in NHI Management Group’s Analysis of Claude Code Security, which shows why code-facing agents need traceable execution evidence. These controls tend to break down when tool chains span multiple services with inconsistent logging because the evaluator cannot reliably reconstruct state.
Common Variations and Edge Cases
Tighter trajectory evaluation often increases engineering overhead, requiring organisations to balance test coverage against trace collection cost and environment complexity. That tradeoff is real, but guidance is evolving toward richer checks for any agent that can act, not just talk.
Some teams use hybrid scoring, where the final response still matters for user experience while trajectory and state assertions determine whether the run is safe. That is usually a better fit than treating the last message as the whole truth. The same applies when an agent has read-only tools versus write access. Read-only agents can still leak sensitive data or select inappropriate sources, but write-capable agents create much higher risk because silent side effects become operational incidents.
There is no universal standard for this yet, but current guidance suggests using the strongest evaluation depth wherever the agent can modify state, access secrets, or chain multiple tools. If the environment lacks reliable state capture, the safest assumption is that final-response scoring is incomplete by design. In those cases, testing should fail closed until the trace can be audited end to end.
In production, final-answer-only evals break down most often when a successful narrative masks an incomplete transaction, especially in systems that retry automatically or reconcile state asynchronously.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A3 | Covers unsafe tool use and hidden execution paths in agent evaluations. |
| CSA MAESTRO | T1 | Focuses on agent trajectory risk and runtime control validation. |
| NIST AI RMF | Requires measurement and monitoring of AI behavior across the lifecycle. | |
| OWASP Non-Human Identity Top 10 | NHI-06 | Relates to insufficient visibility into non-human identity activity and access paths. |
| NIST CSF 2.0 | DE.CM-8 | Supports monitoring of system activity and anomalies during execution. |
Evaluate tool calls and side effects, not just the final answer, to catch unsafe agent behavior.
