AI Production Acceptance Criteria: Agent Go-Live Gates

Most AI agents reach production because a demo worked and a deadline arrived, not because anyone agreed in advance what "secure enough to go live" means. Acceptance criteria fix that: a written, testable set of conditions an agent must meet, with named evidence and a named approver, before it receives production credentials, real data or real users. This guide sets out a practical acceptance gate for AI agents and LLM applications: the criteria to include, the evidence that proves each one, how to tier the gate by risk, how to run a staged go-live with a rollback path, and how the criteria map to NIST, OWASP, the EU AI Act and ISO frameworks. It is for security architects, AI platform owners, risk owners and change approvers. It sits between the AI Agent Risk Assessment Template, which decides how risky an agent is, and the Release Management Security Gates guide, which covers the general mechanics of pipeline approvals. Sources were checked on 8 November 2026.

Key takeaways

Write the criteria before the build, not at the go-live meeting. Criteria agreed late get negotiated down; criteria agreed early shape the design.
Every criterion needs evidence and an owner. "Prompt injection tested" is not a criterion; "critical injection cases pass in the release suite, results attached" is.
Identity is the gate most often skipped. An agent should not go live without its own identity, scoped short-lived credentials and an owner who can revoke them.
Tier the gate by what the agent can do. A read-only internal assistant and an agent that moves money need different bars, set by the risk assessment.
Going live is a stage, not a switch. Shadow mode, limited rollout and a tested kill switch turn acceptance into something you can reverse.

What acceptance criteria are, and what they are not

An acceptance criterion is a pass or fail condition with defined evidence. Taken together, the criteria form the gate an AI system passes once before first production use and again after any material change, such as a new model, a new tool or a wider scope.

Artefact	Question it answers	When	Owner
Risk assessment	How much could go wrong, and which tier is this agent?	Design	Business owner with security
Security evaluations and red teaming	Does the agent resist known and new attacks?	Build and pre-release	Engineering and security testing
Acceptance criteria (this guide)	Is the evidence good enough to let it act in production?	Go-live and material change	Risk owner, signed by security
Release gates in the pipeline	Can this specific build be deployed?	Every deployment	Platform engineering
Runtime monitoring	Is it still behaving as accepted?	Continuously	Operations and SOC

The NCSC's Guidelines for secure AI system development (November 2023) organise the lifecycle into secure design, secure development, secure deployment, and secure operation and maintenance, with responsible release as part of deployment. Acceptance criteria are the checkpoint where the first two areas hand over to the last two.

The acceptance criteria catalogue

Use this catalogue as the starting set. Each row is a criterion with the evidence that satisfies it. Drop rows only with a recorded reason.

Area	Criterion	Evidence
Ownership	Named business owner and technical owner; agent recorded in the AI inventory	Inventory entry with owners, purpose and tier
Identity	Agent runs under its own identity, not a human's account or a shared service account	Identity record; sample of audit logs showing the agent identity
Credentials	Short-lived, scoped credentials issued at run time; no static keys in prompts, code or environment files	Secrets scan results; credential issuance configuration
Authorisation	Each tool and data source granted at least privilege; user-delegated actions carry the user's permissions, not the agent's	Permission matrix reviewed by the data owner
Actions	High-impact actions (payments, deletions, external messages, permission changes) require human approval or are blocked	Tool policy; test showing approval prompt fires
Testing	Critical security evaluation cases pass with repeated attempts; red team findings rated high are fixed or formally accepted	Evaluation report with versions; red team report and closure evidence
Input handling	Untrusted content (documents, web, email, tool results) is marked and cannot change the agent's goals or permissions	Indirect injection test results
Output handling	Model output is validated before it reaches interpreters, databases or browsers	Code review and output handling tests
Supply chain	Model, frameworks, MCP servers and plugins recorded with versions and provenance	AI-BOM or equivalent component list
Data	Data classes the agent can read and write are documented; sensitive data is masked or excluded where not needed	Data flow diagram; data protection review where personal data is involved
Logging	Prompts, tool calls, decisions and identity used are logged with the model identifier, and logs reach the SOC	Sample log records; SIEM ingestion confirmed
Containment	A tested way to stop the agent and revoke its credentials within an agreed time	Kill switch test record with time to stop
Response	Incident playbook covers this agent; on-call owner named	Playbook link; on-call rota
Change control	Model, prompt, tool and permission changes re-trigger the relevant criteria	Pipeline configuration; change policy

Identity and access criteria in detail

Identity criteria deserve their own review because agents fail them quietly: a prototype built with a developer's token often keeps that token in production. OWASP's Top 10 for Agentic Applications for 2026 (published 9 December 2025) lists ASI03 Identity and Privilege Abuse alongside ASI01 Agent Goal Hijack and ASI02 Tool Misuse and Exploitation, and the three combine: a hijacked goal plus broad credentials turns a prompt into an incident.

Distinct identity: the agent has a workload or agent identity that appears in logs, separate from any human. See the Agent Identity and Access Guide.
No standing secrets: credentials are issued per session or per task and expire. Long-lived API keys are an exception that needs a recorded rotation schedule. See the AI Agent Credential Lifecycle and Rotation Guide.
Scopes match the task list: list the tasks the agent is accepted for, then check that every granted scope maps to one. Unmapped scopes are removed before go-live.
Delegation is explicit: where the agent acts for a user, the user's own authorisation is checked for each action, so the agent cannot reach data the user could not.
Recertification date set: the agent's access gets a review date at acceptance. See Agent Identity Recertification.

Tiering the gate by risk

One gate for every agent either blocks low-risk tools or waves through high-risk ones. Let the risk assessment set the tier, and let the tier set the bar.

Tier	Typical agent	Testing bar	Approval
1: Low	Internal, read-only, no sensitive data, no external actions	Standard evaluation suite; no open high findings	Technical owner plus security review of the catalogue
2: Medium	Reads sensitive data or acts inside one system with reversible actions	Full suite with repeated attempts; indirect injection tests on every input channel	Risk owner and security sign-off
3: High	Irreversible or external actions, financial or customer impact, multi-agent chains, code execution	Full suite plus independent red team; human approval on high-impact actions tested	Risk owner, security and, where relevant, data protection and legal
Regulated	Systems in a regulated category, such as high-risk systems under the EU AI Act	Tier 3 plus the evidence the regime requires	As tier 3 plus compliance

For high-risk systems, the EU AI Act's Article 15 expects an appropriate level of accuracy, robustness and cybersecurity throughout the lifecycle, with accuracy metrics declared in the instructions for use and resilience against attacks such as data poisoning, model poisoning, adversarial examples and confidentiality attacks. Article 14 expects human oversight that lets people understand the system's limits, override its output and interrupt it so it comes to a safe halt. The application dates of these high-risk obligations were amended by Regulation (EU) 2026/1744 (the Digital Omnibus on AI), in force since 27 July 2026; secondary summaries give 2 December 2027 for Annex III systems and 2 August 2028 for AI in products covered by EU product legislation. Confirm the dates against the official text before planning to them. See the EU AI Act High-Risk AI Systems Guide.

Small versus large organisations: a small team can run the gate as a single review meeting with the catalogue as a checklist and evidence linked in one ticket. A large organisation with many agents benefits from a standard evidence pack per tier, automated checks for the machine-verifiable rows (secrets scans, evaluation results, logging), and a review board only for tier 3.

Running the gate and the staged go-live

Agree criteria at design: once the risk assessment sets the tier, copy the catalogue for that tier into the project and record any rows dropped with reasons.
Collect evidence as you build: evaluation reports, permission matrices and log samples are attached as they are produced, with the model, prompt and tool versions they apply to.
Pre-acceptance review: security checks the evidence against each row. Gaps become actions with owners, not discussion points.
Shadow mode: the agent runs on production inputs but its actions are logged and not executed, or are executed only in a mock. Compare its proposed actions with what a human did.
Limited rollout: a small user group or a share of traffic, with tighter rate limits and human approval on all write actions. Watch the metrics below daily.
Formal acceptance: the risk owner signs, with any accepted residual risks listed and each given an expiry date.
General availability with rollback ready: keep the kill switch tested and the previous configuration (model version, prompt, tool policy) deployable, so rollback is a configuration change, not a rebuild.

Conditional acceptance is legitimate when it is explicit: "accepted for read-only use until the approval workflow passes testing" is a clear decision. "Accepted, we will fix it later" with no date is not. See the Security Exceptions and Risk Acceptance Guide for how to record it.

When acceptance must be repeated

Acceptance applies to a configuration, not to an agent name. Re-run the relevant rows when any of these change:

Model: a new model or version, including a provider-side update to an alias. Re-run testing rows at minimum.
Tools and permissions: any new tool, scope or data source. Re-run identity, authorisation, action and testing rows.
Autonomy: removing a human approval step, raising rate limits or adding agent-to-agent delegation. Treat as a tier review.
Audience: moving from internal users to customers or the public. Treat as a new acceptance.
Time: annual re-acceptance for tier 2 and above, even with no change, because threats and the surrounding systems move. For the regression suite that supports this, see AI security regression testing.

Detection and measurement

Acceptance assumes certain behaviour; monitoring confirms it. At acceptance, agree the signals that would trigger a review:

Tool call denials and approval rejections: a rise suggests the agent is attempting actions outside its accepted task list.
Use of canary credentials or data: any use is an incident.
Guardrail block rate: sudden changes suggest a new attack pattern or a model change.
Identity anomalies: the agent's credentials used from unexpected locations or outside its run windows.

Metrics that show the gate itself is working:

Share of production agents with a completed acceptance record for their current configuration.
Share of production agents running under their own identity with short-lived credentials.
Number of accepted residual risks past their expiry date.
Kill switch test age: days since each tier 2 and 3 agent's stop and revoke procedure was last exercised.
Post-go-live security findings that an acceptance row should have caught.

The AI Agent Behaviour Baselining and Anomaly Detection Guide covers the runtime side.

Framework mapping

Framework	Where acceptance criteria fit
NIST SP 800-53 Rev. 5	CA-2 Control Assessments; CA-6 Authorization; SA-11 Developer Testing and Evaluation; CM-4 Impact Analyses; AC-6 Least Privilege
NIST CSF 2.0	GV.RM-03 risk management in enterprise risk processes; PR.PS-06 secure development practices; ID.IM-02 improvements from tests and exercises
NIST AI RMF 1.0	Govern, Map, Measure and Manage functions; acceptance is the Manage decision informed by Measure
OWASP Top 10 for Agentic Applications for 2026	ASI01, ASI02 and ASI03 as minimum test and control areas for agents
EU AI Act	Article 14 human oversight and Article 15 accuracy, robustness and cybersecurity for high-risk systems
ISO/IEC 27001:2022	Annex A 8.29 Security testing in development and acceptance; 8.32 Change management
AIUC-1	Certification standard for AI agents covering data and privacy, security, safety, reliability, accountability and society, with pre-deployment testing and periodic retests

Common failure patterns

The prototype credential goes live: the developer token used in the demo becomes the production credential, with the developer's full access.
Testing the model, accepting the agent: a jailbreak benchmark passes, but no one tested what the agent does with its tools when a retrieved document instructs it.
Acceptance without a stop button: the kill switch is designed but never exercised, and on the day it takes hours to find who can revoke the credentials.
One-time acceptance: the agent is accepted with two read-only tools and later gains write access with no re-review.
Residual risks with no expiry: conditional acceptances that become permanent because nobody tracks them.
Logs without identity: prompts are logged but not which identity performed each tool call, so an investigation cannot separate agent actions from user actions.

Practitioner checklist

Set the agent's tier from a completed risk assessment before build starts.
Copy the acceptance catalogue for that tier into the project and record reasons for any dropped rows.
Confirm the agent has its own identity, short-lived scoped credentials and a recertification date.
Require human approval or blocking for payments, deletions, external messages and permission changes.
Attach versioned evaluation and red team evidence; close or formally accept every high finding.
Confirm logs capture prompts, tool calls, identity and model identifier, and reach the SOC.
Exercise the kill switch and credential revocation, and record the time taken.
Go live through shadow mode and limited rollout, with the previous configuration ready to restore.
Give every accepted residual risk an owner and an expiry date.
Re-run acceptance on model, tool, permission, autonomy or audience changes, and annually for tier 2 and above.

AI production acceptance criteria: security gates before an agent goes live