Why do AI agents need step-level evaluation as well as end-to-end testing?

Why This Matters for Security Teams

Step-level evaluation is about seeing the agent’s decisions before they are hidden by a seemingly correct final outcome. In agentic systems, the risky moment is often not the answer itself, but the path taken to get there: a tool call with excessive scope, a malformed prompt to an external service, or a silent misuse of retrieved context. That is why current guidance from the NIST AI Risk Management Framework is better applied to the full lifecycle of agent behaviour, not just final outputs.

Security teams often over-trust end-to-end test suites because they confirm the workflow finished and the user got an answer. That misses failures that matter operationally: unnecessary data exposure, wrong tool selection, bypassed approvals, or weak recovery from invalid inputs. This is especially important for agents that can act across SaaS, code, ticketing, and knowledge systems, where one poor intermediate choice can create persistence, exfiltration, or privilege abuse opportunities. The OWASP Agentic AI Top 10 frames these issues as concrete application risks rather than abstract model concerns.

In practice, many security teams encounter agent failure only after a seemingly successful workflow has already touched the wrong system, exposed the wrong data, or triggered the wrong action.

How It Works in Practice

Step-level evaluation breaks an agent run into observable checkpoints, then scores each decision against expected behaviour. That usually includes whether the agent correctly classified the task, selected an appropriate tool, passed safe parameters, interpreted the tool response accurately, and decided whether to continue, retry, or stop. This is different from end-to-end testing, which only asks whether the final user-visible result is acceptable.

A practical evaluation stack usually combines deterministic checks with human review for higher-risk paths:

• Tool-choice evaluation: did the agent call the right tool for the task, or did it invent a shortcut?
• Argument validation: were the parameters complete, bounded, and free from prompt injection or unsafe instructions?
• Context handling: did the agent use retrieved content faithfully, or overgeneralise from untrusted input?
• Action gating: did the agent seek approval before executing a sensitive action?
• Outcome tracing: can each step be replayed and attributed for audit or incident response?

That approach aligns well with threat modelling in the CSA MAESTRO agentic AI threat modeling framework and with adversarial testing patterns in the MITRE ATLAS adversarial AI threat matrix. It also helps teams detect problems that end-to-end testing can miss, such as an agent correctly completing a request after first attempting a dangerous file read, over-broad API query, or unauthorized chain-of-thought exposure. The point is not to reject every imperfect intermediate step, but to distinguish tolerable variation from unsafe decision-making.

These controls tend to break down when the agent has many external tools, long planning chains, or non-deterministic retrieval paths because the number of valid step sequences becomes too large to evaluate reliably with simple pass-fail tests.

Common Variations and Edge Cases

Tighter step-level evaluation often increases test maintenance and review overhead, so organisations have to balance assurance against development speed. Best practice is evolving here, and there is no universal standard for how granular every workflow should be.

For low-risk internal assistants, coarse step checks may be enough if the agent has no write access and no sensitive data paths. For customer-facing or privilege-bearing agents, the bar should be higher: risky steps deserve dedicated assertions, replay logs, and exception handling tests. The strongest patterns separate benign conversation quality from operational safety, because a fluent answer can still come from unsafe reasoning.

Some edge cases deserve special attention. If an agent uses retrieval-augmented generation, step evaluation should inspect whether retrieved content was relevant, fresh, and trusted before it influenced the next action. If tool outputs are partially structured, the evaluator should check that the agent did not overfit to a malformed field or ignore an error signal. And where agents can chain into code execution, ticket closure, or access changes, step-level checks should focus on approvals and boundaries, not just linguistic quality. Recent industry reporting on real-world misuse, including the Anthropic report on AI-orchestrated cyber espionage, shows why intermediate misuse deserves the same attention as final output quality.

Related resources from NHI Mgmt Group

• Why do AI agents and tool-connected LLMs need runtime controls as well as testing?
• Why do AI agents need runtime evaluation in addition to CI testing?
• Why do AI agents require continuous access evaluation?
• What is the difference between tool-level access and data-level access for AI agents?