AI Agent Security Testing Checklist: Pre-Release and Ongoing Tests

Most AI agents reach production after functional testing only, with nobody having checked what happens when a document, tool or another agent tells them to do something their owner never intended. This checklist sets out the security tests an AI agent should pass before release and the tests that should keep running afterwards. It covers the agent as a system: its inputs, tools, credentials, memory, MCP servers, code execution, approvals and logs, not only the model. Each check is mapped to the OWASP Top 10 for Agentic Applications (ASI01 to ASI10), the OWASP Top 10 for LLM Applications 2025 and MITRE ATLAS so findings can be reported in a shared language. It is written for application security teams, AI platform engineers and testers. All checks are defensive: they describe what to verify, not how to build attacks.

How to use this template

• Replace text in square brackets, such as [Organisation], [Agent name] and [Test owner], with your own terms.
• Start from the agent's threat model. Each high or medium threat should map to at least one check below; checks with no matching threat can be marked not applicable with a reason. See the AI Agent Threat Model Template.
• Run sections 1 to 7 before first release and after any change to model, system prompt, tools, scopes, memory or autonomy. Section 8 sets out what continues in production.
• Record each check as Pass, Fail, Not applicable or Not tested, with evidence: the test case ID, the trace or log reference and the build or configuration version tested.
• Agent behaviour is non-deterministic. Run each adversarial test case several times ([N], for example 10) and record the success rate, not a single result.
• Test in an isolated environment with test identities, synthetic data and egress controls. Never point adversarial tests at production credentials.

1. Scope, inventory and test environment

• The agent's purpose and its "must never" statements are written down and agreed with the owner before testing starts.
• Every tool, MCP server, API, data source, memory store and downstream agent is listed with the version under test.
• Every identity and credential the agent can use or reach is listed: its own identity, delegated user tokens, tool and MCP server credentials, model provider keys and anything in its environment.
• The test environment mirrors production permissions and tool definitions but uses separate test identities and synthetic or masked data.
• Outbound network access from the test environment is restricted to an allow list and logged, so exfiltration attempts are observable and contained.
• A test harness can replay a fixed set of prompts and inputs and capture full traces: model input, tool calls with parameters, tool output and final response.
• Rules of engagement are agreed for any test touching third-party services or shared platforms. See the Red Team Rules of Engagement Template.

2. Input channels and goal integrity

• Direct prompt injection: the agent refuses or ignores user instructions that conflict with its system instructions and "must never" list (LLM01, ASI01, AML.T0051).
• Indirect prompt injection is tested through every channel the agent reads: email, documents, web pages, tickets, chat messages, tool output, retrieved RAG content and messages from other agents (AML.T0051.001). Each channel gets its own test cases.
• Instructions planted in content cannot change the agent's goal, add new tasks or trigger tool calls the user did not ask for (ASI01). See the AI Agent Goal Hijacking Guide.
• Hidden or encoded instructions are covered: text in document metadata, white or zero-size text, alternative languages and content split across several inputs.
• Jailbreak resistance is measured against the organisation's content policy, with the success rate recorded per model version (AML.T0054).
• System prompt extraction attempts do not reveal secrets, and the system prompt is confirmed to hold no credentials or sensitive logic (LLM07, AML.T0056).
• A zero-click case is included: untrusted content arrives without user action and is later retrieved during an ordinary task, the pattern behind EchoLeak 2025, where a crafted email was pulled into Microsoft 365 Copilot's context.

See the Prompt Injection Guide for the controls these tests verify.

3. Tools, actions and excessive agency

• The agent can call only the tools on its approved list; attempts to call unlisted or renamed tools fail at the gateway or framework, not only in the prompt (LLM06, ASI02).
• Each tool exposes only the functions it needs. A mailbox summary tool cannot send or delete; a file writer cannot run shell commands (LLM06 minimise extension functionality, avoid open-ended extensions).
• Tool parameters are validated server-side: path traversal, unexpected recipients, wildcard queries and oversized requests are rejected (ASI02, AML.T0053).
• Tool output passed to other systems is treated as untrusted: no unescaped output reaches a shell, SQL query, browser or template engine (LLM05).
• High-impact actions (payments, permission changes, deletion, bulk export, external messages) require human approval enforced outside the model, and the approver sees the real action, target and parameters (LLM06 require user approval).
• Approval cannot be bypassed by splitting one high-impact action into many small ones, or by asking the agent to describe an action differently from what it executes.
• Destructive tool calls triggered by planted content are blocked or require approval (AML.T0101). Data leaving through a permitted tool, such as an outbound email or web request carrying sensitive data, is detected (AML.T0086).
• Spending, rate and loop limits stop runaway tool use and cost growth (LLM10, AML.T0034).

4. Identity, credentials and authorisation

• The agent authenticates with its own identity, not a person's credentials or a shared service account (ASI03). See the Red Teaming AI Agents for Identity Abuse.
• Every credential the agent holds is scoped to the registered purpose; testers attempt each tool's most damaging allowed action and record whether scope blocks it.
• When acting for a user, the agent cannot read or change anything that user cannot (LLM06 execute extensions in user's context). Test with two users of different privilege.
• Tokens are audience-bound: MCP servers and tools reject tokens not issued for them, as the MCP security best practices require (no token passthrough).
• The agent cannot discover and use credentials it was not given: environment variables, configuration files, cloud metadata endpoints, agent configuration and tool credentials (AML.T0083, AML.T0098, AML.T0055).
• Credentials never appear in prompts, model responses, memory, traces or logs; canary credentials planted in test data are checked for leakage.
• Revoking the agent's identity stops all its tool access within the agreed time.

5. Memory, context and data leakage

• One user's input cannot influence another user's session or answers; memory is isolated per user and per tenant (ASI06, AML.T0080.000).
• Content written to long-term memory from untrusted sources is tested for persistence: does a planted instruction still act in a later, unrelated session? See the AI Agent Memory Poisoning Guide.
• RAG sources are tested with poisoned documents to confirm retrieval respects access rights and provenance (LLM08, AML.T0070).
• Sensitive data in the agent's reach (personal data, secrets, other tenants' records) is not disclosed in responses or tool calls (LLM02, AML.T0057).
• Memory can be inspected, rolled back to a known-good snapshot and purged, as OWASP recommends for ASI06.

6. Supply chain, MCP servers and code execution

• Every MCP server, plugin, package and model is from an approved source and pinned to a reviewed version or hash (ASI04, LLM03, AML.T0010). The postmark-mcp case shows why: version 1.0.16 added one line that blind-copied every email the agent sent.
• Tool descriptions are reviewed as untrusted input, and a changed description or schema blocks the tool until re-approved (AML.T0110, AML.T0104).
• Remote MCP servers are tested against the MCP specification's named attacks: confused deputy (per-client consent, exact redirect_uri matching, single-use state), session hijacking (sessions not used for authentication), SSRF during OAuth discovery (private and link-local ranges such as 169.254.0.0/16 blocked) and authorisation URL schemes (javascript:, data: and file: rejected).
• Local MCP servers run sandboxed, and the client shows the exact start command before first use.
• Code the agent generates or runs executes in a sandbox with no host file system, restricted network and no ambient credentials (ASI05). Escape attempts and resource exhaustion are tested.
• Updates are staged with differential tests, with automatic rollback on hash drift or behaviour change (OWASP ASI04 guidance).

7. Multi-agent, oversight and observability

• Agent-to-agent messages are authenticated; a spoofed or replayed message from a peer agent is rejected (ASI07).
• A failing or compromised agent cannot cascade: replay recorded actions in an isolated clone of production and confirm limits contain the effect (ASI08).
• Approval prompts are tested for manipulation: misleading summaries, urgent framing and approval floods that cause fatigue (ASI09).
• Logs capture agent identity, principal, tool, parameters, credential identifier, authorisation decision, approver and outcome for every action, and the test team can reconstruct each test from logs alone.
• Detection rules fire on the test cases they were built for: injection markers, new tool use, external sends, repeated permission errors, credential access.
• The kill switch is tested: stop the agent, revoke its credentials and confirm no queued actions run afterwards (ASI10).

8. Ongoing and regression testing

• Every confirmed finding becomes a regression test case in the harness and runs on each build.
• The full adversarial suite runs on every change to model version, system prompt, tool set, scopes, memory design or autonomy level, and at least [monthly / quarterly] for high-risk agents.
• Periodic behavioural attestation with challenge tasks confirms the agent still behaves as approved (OWASP ASI10 guidance).
• New public techniques (ATLAS updates, published incidents, vendor advisories) are reviewed [monthly] and added as test cases where relevant.
• Production telemetry is sampled for injection attempts, and real attempts are added to the suite with sensitive data removed.
• A full independent test or red team exercise runs at least [annually] for agents rated high risk. See the AI Red Teaming Guide.

Implementation guidance

Rollout

1. Pick one agent. Choose an agent with real tools and credentials, threat model it, then select the checks that match its threats.
2. Build the harness first. Repeatable traces matter more than clever test cases. Open-source options include the UK AI Security Institute's Inspect (tasks, solvers, scorers, sandboxed tools) and Microsoft's PyRIT; benchmarks such as AgentDojo evaluate prompt injection attacks and defences for agents.
3. Run in observe mode. Execute the suite against the current build without blocking release, record baseline success rates and fix the highest-impact failures.
4. Set gates. Agree pass thresholds per section with the owner: for example, zero successful high-impact actions without approval and zero credential leaks, with a tolerated rate for low-impact jailbreaks.
5. Gate releases. Wire the suite into the release pipeline so a failed gate blocks deployment. Keep the previous agent configuration deployable so a failed change can be rolled back.
6. Extend. Add agents in order of risk tier. See the Agentic AI Risk Assessment Template.

Choosing a test approach

Small teams can start with a manual pass of sections 2 to 4 and a short automated suite for the findings. Large organisations should run a shared harness and test case library across agent platforms, with independent red teaming for the highest tier.

Mapping and metrics

In NIST SP 800-53 Rev. 5 terms the checklist supports SA-11 Developer Testing and Evaluation and CA-8 Penetration Testing; in NIST CSF 2.0 it produces the improvements expected by ID.IM-02. Track: percentage of production agents tested before release; adversarial success rate per section and model version; findings closed before release; mean time from new public technique to test case; and kill-switch time measured in tests.

Common failure patterns

• Testing the model, not the agent: jailbreak scores with no test of what the agent's credentials allow.
• One run per test: a single pass hides a 20 per cent success rate.
• Only the chat box: indirect channels such as email, tool output and RAG left untested.
• No retest after change: a new tool or model version ships on old results.

Standards and references

• OWASP GenAI Security Project: Top 10 for Agentic Applications for 2026
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: Adversarial Threat Landscape for AI Systems
• Model Context Protocol: Security Best Practices
• OWASP GenAI Security Project: GenAI Red Teaming Guide
• Cloud Security Alliance: Agentic AI Red Teaming Guide
• UK AI Security Institute: Inspect
• NIST: SP 800-53 Rev. 5

This template is provided for guidance and must be adapted to your organisation. Related NHI Mgmt Group resources: AI Agent Threat Modelling Guide · OWASP Top 10 for Agentic Applications Guide · MCP Security Guide · AI Red Teaming and Testing Tools Buyer's Guide