They should do that whenever a workflow can partially succeed, fail after acceptance, or change between runs. A repeatable test must hold the model, instructions, harness, and environment steady, then exercise recovery cases such as permission changes or lost responses. That separates true execution quality from luck, shared state, or hidden assistance from earlier runs.
When success rate stops being the right unit of measure
Treat agent reliability as a scenario and recovery problem when the workflow can get partway to a valid outcome, then stall, drift, or depend on hidden state from a prior run. A single success percentage hides whether the agent can retry safely, recover from partial completion, or behave consistently when permissions, responses, or environment conditions change.
That shift matters because many agent workflows are not binary. They can produce useful output, commit side effects, or consume budget before failing, which means the real question is not only whether the task eventually “succeeds”, but whether it fails in a bounded, observable, and recoverable way.
What a scenario-based test actually needs to hold steady
A useful reliability test keeps the model, instructions, harness, and environment fixed, then changes one condition at a time. That lets you separate execution quality from luck, cached context, shared state, hidden retries, or assistance left behind by an earlier run.
The scenarios should reflect the real failure surfaces of the workflow, not just a happy-path benchmark. If permissions can change, responses can disappear, tools can time out, or a task can partially complete, each of those conditions should be exercised explicitly so you can see whether the agent recovers cleanly or quietly degrades.
- Partial success: the agent completes one step, then fails before finishing the workflow.
- Post-acceptance failure: the task is accepted or queued, but later cannot be completed.
- Environment drift: the same prompt behaves differently because state, access, or tool availability changed.
- Recovery path: the agent retries, resumes, rolls back, or escalates in a predictable way.
That is also why an isolated run is rarely enough. If a second run benefits from prior state, the metric may improve while the underlying system becomes less trustworthy. The better measure is whether the system can reproduce the same outcome under the same conditions, and handle the same failure mode in a controlled way when the conditions change.
Why recovery behavior matters more than a headline metric
Teams should care about how the agent behaves after failure because that is where operational risk usually appears. A workflow that is 90 percent successful but leaves stale side effects, duplicated actions, or ambiguous state on failure is often worse than a workflow with a lower headline score but clean rollback and clear recovery.
Reliable agent evaluation should therefore include AI Agent Observability, Audit and Incident Response Guide style thinking: can you attribute what happened, detect when the agent went off track, and recover without guessing? It should also include Zero Trust for AI Agents discipline, because a recovery test is only meaningful when access, privilege, and action boundaries are still enforced during retries and exceptions.
For agentic workflows, the same issue appears in tool use and delegated action. If the agent can partially execute a task, then a missing response, changed permission, or delayed approval can alter the outcome more than the prompt itself. A robust test suite should make that visible instead of folding it into an average success rate. AI Agent Authorisation Guide is useful here because it frames per-action permission as part of correctness, not just security.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI08 — Cascading Failures | Partial completion and retry failures can propagate through agent workflows. |
| ASI03 — Identity & Privilege Abuse | Recovery tests must respect changing access and per-action authorization. | |
| Recommendation — Test recovery paths to prevent one failed step from cascading into broader task failure. Verify that retries and exception paths still enforce per-action authority. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Scenario-based reliability needs observable evidence for what happened during partial or failed runs. |
| AC-6 — Least Privilege | Permission changes and bounded access are central to recovery-oriented testing. | |
| Recommendation — Log agent actions so failed or resumed runs can be reviewed and reconciled. Limit agent access so recovery tests show controlled behavior under reduced privilege. | ||
| NIST CSF 2.0 | DE.CM-01 — The network is monitored to detect potential cybersecurity events | Agent reliability tests benefit from monitoring that reveals abnormal execution and recovery behavior. |
| Recommendation — Monitor agent runs for failed tools, retries, and inconsistent execution states. | ||
Practitioner Guidance
What to verify: Check whether the agent can resume, retry, or fail closed after a partial side effect. If the workflow leaves behind files, tickets, messages, or approvals, validate the cleanup or reconciliation path as part of the test, not after the fact.
Decision rule: If two runs can produce different outcomes because one run had an advantage from earlier state, treat the metric as contaminated. In that case, measure scenario coverage and recovery quality before you trust any aggregate success rate.
What good looks like: The agent’s result is stable under a fixed harness, and its failure modes are predictable when you remove access, interrupt a tool call, or force a retry. The important signal is not perfection, it is controlled behavior under stress.
Practitioner takeaway: Reliability for agents is an execution-and-recovery property, not a single scoreboard number. When state, privilege, or external dependencies can change mid-run, test the failure path as deliberately as the success path.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 30, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
