AI Red Teaming vs Penetration Testing: How to Scope, Contract and Accept Each

Organisations commissioning security testing for an AI product often buy a penetration test and assume it covers the model, or buy an AI red team and assume it covers the API, and discover the gap only when the report arrives. This guide is for security leads, procurement and assurance teams who write the statement of work, agree the rules of engagement and accept the deliverables. It compares the two engagements on what they are scoped against, how findings behave, who should test, what the report must contain and how each one fails at the scoping stage. It extends the short comparison in the AI Red Teaming Guide and does not repeat its attack categories or tool list. Sources were checked on 13 November 2026.

Key takeaways

• They test different things. A penetration test asks whether known classes of vulnerability can be exploited in the system around the model; an AI red team asks what the AI behaviour itself can be made to do, including harms caused for ordinary users.
• Findings behave differently. A pentest finding is usually deterministic and closed by a code or configuration fix; an AI red team finding is probabilistic, so it must be reported with attempt counts and closed by a regression test, not a single retest.
• Scope by claims, not just by assets. A pentest scope lists hosts, URLs and roles; an AI red team scope also lists model and prompt versions, tools, data sources, the harm taxonomy and the actors to emulate.
• Contract for different deliverables. Ask the pentest supplier for severity-rated, reproducible vulnerabilities; ask the AI red team for a harms list, raw input and output records, success rates and the regression cases that keep the result alive.
• Most AI products need both, in one plan. Shared scope, shared test identities and one remediation tracker stop each team assuming the other covered the boundary between application and model.

The short answer

A traditional penetration test is, in the words of the UK NCSC, a method for gaining assurance in the security of a system, and it is not the primary way to find vulnerabilities. It validates that the system is not vulnerable to known issues on the day of the test. NIST SP 800-115 frames it within planned technical testing whose findings feed mitigation. The target is the web application, API, identity layer, cloud and infrastructure, and the output is a list of vulnerabilities with a severity and a fix.

AI red teaming grew from the same adversarial tradition, but Microsoft's guidance notes that the term has extended beyond traditional cybersecurity to cover many kinds of probing, testing and attacking of AI systems, and that both benign and adversarial usage can produce harmful outputs. It tests the model, the application that wraps it and any agent with tools, for security failures and for content and behaviour harms. CISA treats AI red teaming as part of AI testing, evaluation, verification and validation (TEVV) and argues the differences from software testing are "none so large as to warrant a drastically different approach". For a buyer, the commissioning discipline of a pentest still applies; the scope, evidence and acceptance criteria change.

Penetration testing: threat model, scope and failure modes

Threat model and architecture

The pentest threat model is an attacker who reaches the system through its interfaces and tries to break authentication, authorisation, input handling and configuration. For an AI product, the architecture in scope is everything around the model call: the chat front end, the API gateway, session handling, tenant isolation, the retrieval store and its access controls, the secrets the application uses to call the model provider, and the cloud resources it runs on. The model is treated as a component whose output is untrusted input to the next layer.

Scoping inputs that work

The NCSC advises that scoping involve risk owners, technical staff who know the system and a representative of the test team, and that any current vulnerability assessment be shared with the testers. The agreed plan covers the technical boundaries of the test, the types of test, the timeframe and effort in resource days, tester requirements, compliance needs and the reporting scheme. For an AI application, add these to the usual scope:

• Model-adjacent endpoints: the API that accepts prompts, file upload paths, webhook receivers for tools and any admin interface for prompts or guardrails.
• Retrieval access control: whether a user can retrieve documents belonging to another tenant or role through search, which is an authorisation test even though it surfaces through the model.
• Output handling: whether model output is rendered or executed unsafely downstream, which maps to OWASP LLM05 Improper Output Handling and to ordinary injection testing.
• Credentials: how the application stores and scopes its model provider keys and the tokens its agents use.

Where a pentest falls short on AI

• No harm testing: a tester will not normally assess whether the assistant gives dangerous advice to a distressed user, because that is not a vulnerability class in a pentest methodology.
• Single-shot evidence: a payload that fails once is often recorded as not exploitable, when a probabilistic system may yield on a later attempt.
• Model treated as a black box: prompt injection that steers an agent into misusing a legitimate tool can pass every authorisation check, because the agent is authorised; only behaviour testing finds it.

AI red teaming: threat model, scope and failure modes

Threat model and architecture

The AI red team threat model includes the pentest attacker but adds two more actors: the benign user who triggers a harm without intending to, and content the system reads, such as documents, email or web pages, that carries instructions. MITRE ATLAS catalogues the latter as AML.T0051 LLM Prompt Injection with the sub-technique AML.T0051.001 Indirect, alongside AML.T0054 LLM Jailbreak. The architecture in scope is the base model and its safety layer, the application with its system prompt and filters, and any agent with memory, tools and credentials. Microsoft advises starting with the base model to understand the risk surface, testing with and without mitigations, and testing on the production interface as much as possible, while recording which endpoint was used.

Scoping inputs that work

• Versioned system definition: model name and version, system prompt version, guardrail configuration, tool list with each tool's permissions, and retrieval sources. A finding is only meaningful against a recorded configuration.
• Harm taxonomy: the harm categories in scope with definitions and examples, agreed with product, legal and responsible AI owners before testing starts.
• Actors and personas: attacker, insider, curious user and vulnerable user, each with a stated goal.
• Claims under test: statements the product team makes, such as "the assistant never reveals another customer's data" or "the agent cannot send email outside the domain". Each test objective maps to a claim.
• Team composition: Microsoft's guidance recommends cross-disciplinary testers, including security specialists and people who bring a regular user's perspective, and assigning specialists to the harms they understand. Its paper on more than 100 products adds that automated scorers are less reliable in specialised areas such as medicine, cyber security and CBRN, so domain experts are needed.

Where an AI red team falls short on infrastructure

• Broken access control behind the chat: an AI red team working through the user interface may never test the API directly, so an insecure direct object reference on the conversation history endpoint goes unnoticed.
• Examples without rates: Microsoft warns that red team examples should not be read as a metric for how pervasive a harm is; without systematic measurement the buyer cannot tell a rare edge case from a common failure.

Rules of engagement compared

Rules of engagement for a pentest are well understood: authorised targets, testing windows, prohibited techniques such as denial of service, emergency contacts and handling of credentials found. AI engagements need those plus several clauses a standard template lacks. Use the Red Team Rules of Engagement Template as the base and add the AI rows below.

For agent tests, containment is a prerequisite: see the AI Agent Sandboxing and Code Execution Guide.

Findings: vulnerability versus harm

The biggest acceptance problem is that the two engagements produce different kinds of finding, and buyers try to force both into one severity scale. A pentest finding is a vulnerability: a weakness, a proof that it can be exploited, a severity and a fix. The NCSC expects the report to give the testers' view of the risk each vulnerability poses and a method of resolving each issue, while risk decisions remain with the organisation.

An AI red team finding may be a vulnerability, for example system prompt leakage (OWASP LLM07) or excessive agency (LLM06), but it may also be a harm with no exploit at all: a benign question that produces a confident and dangerous falsehood. It is also probabilistic. Write the finding record so that it carries:

• Configuration identifier: model, prompt and guardrail versions and the endpoint.
• Input and output: the date, a unique identifier for the input and output pair, the prompt and the output or a screenshot, which is the minimum Microsoft recommends per example.
• Attempts and successes: how many attempts were made and how many succeeded, with paraphrases counted separately.
• Harm category and impact: from the agreed taxonomy, with who would be affected.
• Layer: model, application or agent, which decides the owner of the fix.
• Proposed regression case: the test that will show the issue is fixed and stays fixed.

Keep two severity scales and one tracker. Vulnerabilities keep CVSS or your internal scheme; harms use a product-specific matrix of impact against likelihood, where likelihood is informed by the observed success rate. Both go into the same Penetration Test Remediation Tracker Template with a column for finding type.

Deliverables to contract for

Hand-over artefacts are the deliverable most often missing. Microsoft lists "The work of securing AI systems will never be complete" among its eight lessons, and recommends repeated break-fix cycles. A report without regression cases loses its value at the next model update. For reading a pentest report critically, see How to Read a Penetration Test Report.

Failure modes in scoping

• "AI in scope" in a pentest statement of work. The supplier tests the web layer, adds a handful of prompt injection attempts and reports none succeeded. Name the claims, the harm taxonomy and the attempt counts, or buy a separate AI engagement.
• Unversioned scope. The provider updates the model mid-engagement and half the findings no longer reproduce. Freeze versions or log them per finding.
• Agents tested with real permissions. The scope names the agent but not its identities, so tests run with production credentials. Require test identities in the rules of engagement.
• Same team, same week, no shared tracker. The pentest and red team each assume the other covered retrieval access control. Assign each boundary to one engagement in writing.
• Harm taxonomy written by testers alone. Without product and legal owners the categories miss what matters for the use case. Agree it before day one.
• Accepting "no critical findings" as clearance. For a probabilistic system this describes the attempts made, not the absence of the failure. Ask for attempts per objective.

Decision criteria: which to commission, and when

Small organisations can use one supplier with both skill sets if the statement of work splits scopes and deliverables. Large organisations usually keep pentesting on the existing panel and run AI red teaming through a central AI assurance function, with joint scoping per product. The Penetration Test Scoping Checklist remains the starting point for the application layer.

Worked scenario

A retailer is launching a customer service agent that reads order history through retrieval, can issue refunds up to a limit through a tool, and escalates to staff. The security lead writes one test plan with two engagements.

1. Joint scoping meeting: risk owner, product owner, platform engineers and both suppliers list claims: customers see only their own orders, refunds never exceed the limit, the agent never discloses staff contact details, and the agent does not give unsafe advice about product recalls.
2. Boundary assignment: the pentest owns the order API, session handling, tenant isolation in retrieval and the refund tool's server-side limit check; the AI red team owns prompt injection through order notes and email, refund manipulation through conversation, disclosure through the model and harm categories.
3. Environment: a staging copy with synthetic orders, a refund tool wired to a mock payment service and an agent identity that can refund only test accounts.
4. Execution: the pentest finds the refund limit is enforced only in the prompt, not in the API, a deterministic critical finding. The red team finds that an instruction placed in an order note leads to a refund to a different account in a minority of attempts, and records the rate.
5. Remediation: engineering moves the limit and payee check into the refund service. The red team's case becomes a regression test run 50 times on each release candidate; the pentest supplier retests the API once.
6. Acceptance: the release board signs off on the pentest retest, the regression pass rate at the agreed threshold and a residual risk statement for the harm categories.

Where identity fits

Agents act through non-human identities: API keys for the model provider, OAuth tokens for business systems and service accounts for retrieval. A pentest should check how those credentials are stored and scoped; an AI red team should check what the agent can be induced to do with them. Agree in scope who tests each, and provision separate test identities so a test cannot spend real authority. See the Agentic AI Security vs LLM Application Security comparison for where agent permissions change the threat model.

Detection and measurement

• Boundary coverage: percentage of agreed system claims that are assigned to an engagement and tested. Target 100% for customer-facing products.
• Attempts per objective: median number of attempts per AI red team objective, reported per harm category, so "not reproduced" has meaning.
• Regression conversion: share of AI red team findings that have an automated regression case within 30 days.
• Re-open rate: findings that reappear after a model or prompt change, which shows whether regression cases work.

Framework mapping

Practitioner checklist

• Write the product's security and harm claims and assign each to the pentest, the AI red team or both.
• Add model-adjacent endpoints, retrieval access control and output handling to the pentest scope.
• Give the AI red team a versioned system definition, an agreed harm taxonomy and named actors.
• Extend the rules of engagement with provider permission, version logging, agent stop conditions and tester welfare.
• Provision dedicated test identities with fake, monitored credentials for every agent tool.
• Require attempt counts and success rates for every AI finding.
• Contract for regression cases your evaluation pipeline can run, not only a report.
• Track vulnerabilities and harms in one tracker with separate severity scales.
• Trigger a targeted AI red team round on every model, prompt or provider change.

Standards and references

• NCSC: Penetration testing
• NIST: SP 800-115 Technical Guide to Information Security Testing and Assessment
• NIST: SP 800-53 Rev. 5
• CISA: AI Red Teaming: Applying Software TEVV for AI Evaluations
• UK Government: Code of Practice for the Cyber Security of AI
• Microsoft: Planning red teaming for large language models and their applications
• arXiv: Lessons From Red Teaming 100 Generative AI Products
• OWASP: Top 10 for LLM Applications 2025

Related NHI Mgmt Group resources: AI Red Teaming Guide, Penetration Testing Guide, AI Red Team Test Plan Template, OWASP GenAI Red Teaming Manual Guide, AI Red Teaming vs Evaluations vs Benchmarks.