What breaks when an MCP eval only scores the final response?

Why This Matters for Security Teams

When an MCP eval only scores the final response, it rewards the appearance of success instead of the actual security and operational path taken to get there. That is a serious problem for agentic systems because a tool call can fail quietly, a retry can duplicate state, or an unsafe action can happen even if the last message looks clean. The result is blind spots in incident review, compliance evidence, and production change control.

For teams evaluating OWASP Agentic AI Top 10, the issue is not just model quality. It is whether the evaluation can detect harmful trajectories, hidden exceptions, and side effects that a final answer cannot reveal. NHI Management Group’s research on OWASP Agentic Applications Top 10 treats this as a core control gap, because agent behaviour must be assessed across the whole run, not only at the end.

In practice, many security teams discover duplicate records, leaked secrets, or unauthorized tool use only after downstream systems have already absorbed the damage.

How It Works in Practice

Trajectory scoring evaluates the full chain of agent decisions: prompts, tool selections, tool outputs, intermediate errors, retries, and the final state. State assertions add a second layer by checking whether the environment ended in the expected condition, such as whether a record was created once, permissions stayed unchanged, or a file was not exposed. Final-response scoring cannot see these outcomes unless the last message explicitly reveals them, and agents often do not.

This is especially important in mcp environment because the protocol is about structured tool access, not just conversational output. If the agent calls the wrong tool, retries after a timeout, or writes to the wrong resource, the final answer may still read as successful. The evaluation needs to inspect the execution trace and compare it to intended policy.

• Score the full trajectory, not only the final natural-language response.
• Assert expected post-conditions in the target system after each task.
• Check tool call order, argument values, and retry behaviour for anomalies.
• Flag hidden failures, duplicates, and unauthorized side effects as distinct outcomes.

These controls align with the risk patterns described in OWASP Top 10 for Agentic Applications 2026, especially where tool misuse and unsafe execution paths matter more than the final answer. The broader operational context is also covered in NHI Management Group’s Analysis of Claude Code Security, which shows why code-facing agents need traceable execution evidence. These controls tend to break down when tool chains span multiple services with inconsistent logging because the evaluator cannot reliably reconstruct state.

Common Variations and Edge Cases

Tighter trajectory evaluation often increases engineering overhead, requiring organisations to balance test coverage against trace collection cost and environment complexity. That tradeoff is real, but guidance is evolving toward richer checks for any agent that can act, not just talk.

Some teams use hybrid scoring, where the final response still matters for user experience while trajectory and state assertions determine whether the run is safe. That is usually a better fit than treating the last message as the whole truth. The same applies when an agent has read-only tools versus write access. Read-only agents can still leak sensitive data or select inappropriate sources, but write-capable agents create much higher risk because silent side effects become operational incidents.

There is no universal standard for this yet, but current guidance suggests using the strongest evaluation depth wherever the agent can modify state, access secrets, or chain multiple tools. If the environment lacks reliable state capture, the safest assumption is that final-response scoring is incomplete by design. In those cases, testing should fail closed until the trace can be audited end to end.

In production, final-answer-only evals break down most often when a successful narrative masks an incomplete transaction, especially in systems that retry automatically or reconcile state asynchronously.

Related resources from NHI Mgmt Group

• How can organizations manage the risk of credential leaks in MCP frameworks?
• What challenges do unmanaged API keys pose within MCP?
• What breaks when AI agents are judged only by final benchmark scores?
• What breaks when sensitive data is not inspected before an MCP tool response reaches an AI model?