Set acceptance criteria around workflow reliability, not only task success: the agent should complete the intended steps, use approved tools, preserve state correctly, and fail safely when permissions or data are missing. Include measurable checks for deterministic routing, bounded retries, auditability, and correct escalation to a human when the agent cannot proceed confidently. Production approval should require traceable runs against representative scenarios.
What acceptance criteria should prove before an AI agent goes live?
Acceptance criteria should prove that the agent is dependable in the real workflow, not just that it can finish a single happy-path prompt. Teams should define what “good” means for step completion, tool selection, state handling, retry behaviour, escalation, and evidence of the run. The bar should reflect production conditions, including missing permissions, incomplete data, and ambiguous instructions.
That means the criteria need to describe observable outcomes, not vague confidence statements. A production-ready agent should be able to show which tools it used, which branches it took, where it stopped, and what it did when it could not safely continue. Traceability matters because acceptance is really a decision about operational control, not model optimism.
How do you test workflow reliability instead of task success alone?
Workflow reliability means the agent can move through the intended sequence without breaking state, skipping required checks, or taking an unapproved shortcut. A good acceptance set includes representative scenarios that test normal paths, partial failures, permission denials, and edge cases that force the agent to ask for help. That is especially important for agents that chain tools or act across multiple systems.
Build the checks around deterministic routing where possible, bounded retries where failure is expected, and state preservation across each handoff. The test should show that the agent can retry within policy, stop retrying when further attempts are pointless, and return control cleanly when the current context is not sufficient. If the workflow depends on a human checkpoint, the acceptance criteria should verify that the handoff is explicit rather than implied.
How should teams define safe failure and auditable operation?
Safe failure means the agent does not improvise a risky substitute when permissions, data, or tool access are missing. Acceptance criteria should require the agent to fail closed on sensitive actions, surface the reason for stoppage, and escalate to a human when confidence is too low or the workflow cannot be completed within policy. Auditability should be part of the requirement, not an afterthought.
That is why production approval should include traceable runs against representative scenarios. The logs should make it possible to reconstruct the decision path, confirm which inputs were available, and verify that the agent stayed within the approved action set. For agent designs that depend on delegated authority or approvals, AI Agent Authorisation Guide is useful for framing how per-action policy and human approval translate into testable acceptance conditions.
Risk and Threat Considerations
Acceptance criteria that focus only on output quality can miss the main production risk, which is uncontrolled action. An agent may appear successful while using the wrong tool, overreaching its permissions, or continuing after the environment has changed. The most damaging failures are often workflow failures, because they create hidden side effects even when the final answer looks plausible.
Failure mechanism: The agent passes a task-level test but fails at tool governance, state integrity, or escalation discipline, so it can take unsafe actions, repeat them, or mask the point of failure from reviewers.
Impact: Teams may approve an agent that is operationally brittle, hard to investigate, and capable of making silent mistakes at scale, especially when it interacts with production data or privileged systems. For agent threat patterns, Agentic AI Security Guide is a strong companion for understanding how workflow controls map to real abuse paths.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST SP 800-53 Rev 5 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Acceptance criteria must prevent unsafe privilege use by agents. |
| Recommendation — Test that the agent stays within approved privileges and escalates when access is insufficient. | ||
| NIST SP 800-53 Rev 5 | AU-12 — Audit Record Generation | Traceable runs and reconstructable decisions are central to approval. |
| IA-5 — Authenticator Management | Safe production use depends on controlled credentials and token handling. | |
| AC-6 — Least Privilege | Acceptance should confirm the agent cannot exceed its intended authority. | |
| Recommendation — Generate audit records that capture tool use, decisions, and escalation points. Manage and rotate credentials and tokens used by the agent. Constrain the agent to the minimum permissions needed for the workflow. | ||
Practitioner Guidance
What to verify: Require evidence that the agent can complete the full workflow under both normal and degraded conditions, not just pass isolated prompt tests. Verify the tool sequence, the retry ceiling, the stop condition, the escalation trigger, and the state that is preserved or discarded at each step.
Decision rule: If a scenario can lead to privileged action, external side effects, or irreversible change, the acceptance criterion should require human-reviewable traces and a fail-safe exit path. If the agent cannot show those behaviours consistently, treat it as not production-ready even if its task accuracy is high.
Practitioner takeaway: The right acceptance bar is whether the agent can be trusted to behave predictably when the workflow is messy, incomplete, or partially blocked, because that is where production risk actually appears.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on November 9, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org