AI Red Teaming Guide: Scoping, Methods, Tools and Scoring

Generative AI systems fail in ways that conventional security testing does not look for: a model that can be talked into producing harmful content, an application that leaks its system prompt or other users' data, or an agent that takes an action nobody authorised. AI red teaming is the structured, adversarial probing of models, applications and agents to find those failures before attackers or ordinary users do. This guide covers how to scope an AI red team engagement, the difference between harm testing and security testing, the main attack categories at a high level, where automated tools fit, how to score and report results, what regulators now expect, and how to keep testers and real systems safe. It is for security and responsible AI teams running or commissioning AI red teaming, and for leaders who need to know what good looks like. It deliberately contains no attack prompts or payloads.

Key takeaways

Scope by layer. Test the base model, the application built on it and any agent with tools separately, because each layer fails differently and has different owners.
Red teaming is not benchmarking. Microsoft's AI red team lists "AI red teaming is not safety benchmarking" as one of its eight lessons from more than 100 products; it finds failure modes, while systematic measurement shows how often they occur.
Automate the breadth, keep humans for depth. Tools such as PyRIT, garak and promptfoo generate and score large volumes of probes and catch regressions; creative multi-step attacks and judgement about harm still need people.
Regulators now expect adversarial testing. The EU AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluation "including conducting and documenting adversarial testing", and NIST AI 600-1 lists AI red-teaming among its suggested actions.
Contain the test itself. Testers need wellbeing support, and agent tests need sandboxes, scoped credentials and kill switches, because evaluations of capable agents have already reached real systems.

What AI red teaming covers

Microsoft's guidance notes that the term red teaming "has extended beyond traditional cybersecurity and evolved in common usage to describe many kinds of probing, testing, and attacking of AI systems." In practice there are three test targets, and most programmes need all three.

Layer	What is tested	Typical access	Example findings
Model	The base or fine-tuned model with its built-in safety training and the provider's default filters	API endpoint	Harmful content generation, bias, confabulation, training data leakage, refusal gaps
Application	The product: system prompt, retrieval (RAG), content filters, output handling, user interface, session and tenant boundaries	Production-like UI and API	System prompt leakage, cross-user data exposure, unsafe rendering of output, filter bypass, cost abuse
Agent	An application that plans and calls tools, with memory and credentials	Isolated environment with test identities and tools	Indirect prompt injection leading to actions, privilege misuse, data exfiltration through tools, approval bypass

Microsoft recommends starting with the base model "to understand the risk surface", then testing the application with and without mitigations, and testing on the production UI as much as possible because it "most closely resembles real-world usage." OWASP's GenAI Red Teaming Guide (January 2025) frames the same idea as four areas: "model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis."

Harm testing, security testing and penetration testing

AI red teams test for two kinds of failure, often in the same engagement, and it helps to keep them separate in planning and reporting.

Harm (responsible AI) testing: can the system produce content or behaviour that harms users or third parties, such as dangerous, violent, hateful or abusive content, harmful bias or misinformation? The actor may be benign: Microsoft's threat model ontology states the actor's intent "could be adversarial (e.g., a scammer) or benign (e.g., a typical chatbot user)."
Security testing: can an attacker break confidentiality, integrity or availability through the AI component, for example by extracting data, hijacking an agent's tools or running up cost? Here the actor is adversarial and the impact is measured like any other security finding.

Activity	Question	Output	When to use
AI red teaming	What can go wrong, and how?	Failure modes, reproducible examples, severity	Before launch, on major change, periodically
Safety benchmark or evaluation	How often does a known failure occur?	Rates against a fixed dataset	Every release, for trend and comparison
AI application penetration test	Are the surrounding web, API, identity and infrastructure controls sound?	Conventional vulnerability findings	Alongside red teaming, by application security testers
Guardrail regression testing	Do our filters still catch what they caught last week?	Pass or fail per test case in CI	On every prompt, model or filter change

Decision guidance: an AI penetration test alone misses harm and model behaviour; AI red teaming alone misses ordinary flaws such as broken authorisation on the API behind the chatbot. Commission both, ideally with shared scope, and use guardrail regression tests to keep the results from decaying.

Scoping and rules of engagement

Microsoft's AI red team describes each operation with a five-part ontology, which makes a practical scoping template:

System: "the end-to-end model or application being tested", including model version, system prompt version, tools, data sources and filters. Record which endpoint was tested; findings on a staging endpoint may not hold in production.
Actor: who is being emulated, for example an external attacker, a malicious insider, a curious user or a vulnerable user.
TTPs: the tactics, techniques and procedures in scope, mapped where possible to MITRE ATLAS and the OWASP Top 10 for LLM Applications.
Weakness: "the vulnerability or vulnerabilities in the System that make the attack possible."
Impact: "the downstream impact created by the attack (e.g., privilege escalation, generation of harmful content)." Define in advance which impacts count as critical for this product.

Rules of engagement

Harm taxonomy: agree the harm categories in scope and their definitions. NIST AI 600-1 lists 12 generative AI risks, from "CBRN Information or Capabilities" and "Confabulation" to "Information Security" and "Value Chain and Component Integration", which make a good starting list.
Environment: production-like, with test accounts and synthetic data. Agents must use dedicated test identities with the minimum permissions and no route to real third-party systems.
Prohibited actions: no testing of third-party models or services without their permission, no real personal data, no actions with real-world effect outside the sandbox.
Disclosure: how to report a critical finding immediately, and how to report vulnerabilities in a supplier's model through its disclosure process.
Data handling: harmful outputs collected during testing are sensitive; store them with restricted access and a retention limit.

Attack categories at a high level

The categories below describe what to test, not how. Detailed defences live in the linked guides.

Category	What the test establishes	OWASP LLM Top 10 (2025)
Direct prompt injection and jailbreaks	Whether user input can override instructions or safety training	LLM01 Prompt Injection
Indirect prompt injection	Whether content the system reads (documents, email, web pages, tool output) can steer it	LLM01 Prompt Injection
Sensitive data and system prompt leakage	Whether the system reveals secrets, other users' data or its own instructions	LLM02 Sensitive Information Disclosure; LLM07 System Prompt Leakage
Unsafe output handling	Whether downstream components execute or render model output unsafely	LLM05 Improper Output Handling
Excessive agency	Whether an agent can be induced to take actions beyond its purpose with its tools and credentials	LLM06 Excessive Agency
Retrieval and embedding weaknesses	Whether RAG returns data the user should not see, or can be poisoned	LLM08 Vector and Embedding Weaknesses
Harmful content and misinformation	Whether the system produces content in the agreed harm categories, or confident falsehoods	LLM09 Misinformation
Resource abuse	Whether inputs can drive unbounded cost or denial of service	LLM10 Unbounded Consumption

Microsoft's lesson that "you don't have to compute gradients to break an AI system" matters for prioritisation: the paper adapts the saying to "real attackers don't compute gradients, they prompt engineer." Start with the simple, realistic techniques before research-grade adversarial machine learning. Injection is covered in the Prompt Injection Guide, and model and application controls in the LLM Security Guide. The EchoLeak zero-click prompt injection in Microsoft 365 Copilot shows why indirect channels belong in scope.

Methods: manual, guided and automated

Microsoft's planning guidance recommends open-ended testing first, to "uncover blind spots in your understanding of the risk surface", then building a list of harms with definitions and examples, then guided rounds that probe those harms and test mitigations. Red teamers should be diverse in expertise and background, with "both benign and adversarial mindsets."

Automated tooling

Automation is one of Microsoft's eight lessons ("Automation can help cover more of the risk landscape"), alongside "The human element of AI red teaming is crucial." The open-source tools below are examples, described from their own documentation; none is endorsed.

Tool	Documented purpose	Documented concepts	Best fit
PyRIT (Microsoft, MIT licence)	"The Python Risk Identification Tool for generative AI (PyRIT) is an open source framework built to empower security professionals and engineers to proactively identify risks in generative AI systems."	Microsoft's red team uses it to generate prompt variations, score outputs and automate attack attempts	Scripted, multi-turn campaigns driven by an internal red team
garak (NVIDIA, Apache 2.0)	"Generative AI Red-teaming & Assessment Kit" that "checks if an LLM can be made to fail in a way we don't want"	Probes, detectors, generators, harnesses and evaluators; JSONL run report and a hit log	Broad scanning of a model or endpoint across known failure classes
promptfoo	Red teaming that provides "a quantitative measure of risk before deployment"	Plugins as adversarial testers per failure mode; deterministic and model-graded metrics; reports; CI/CD regression checks	Application-level testing and regression gates in the delivery pipeline

Treat tooling as dual-use: attack libraries and generated outputs belong in access-controlled repositories, and tool runs against shared model endpoints need the model owner's agreement.
Keep humans in the loop: automated scorers misjudge context, and novel multi-step attacks on agents still come mostly from people.

Evaluation and scoring

Model outputs are probabilistic, so a single success or failure proves little. Microsoft's guidance warns: "It is important that people do not interpret specific examples as a metric for the pervasiveness of that harm."

Record every attempt: at minimum the date, a unique identifier for the input and output pair, the input, and a description or screenshot of the output, plus model, prompt and filter versions.
Attack success rate (ASR): the proportion of attempts at a given objective that achieved it. Run each attack several times, report ASR with the number of trials, and compare like with like: same objective, same configuration.
Severity: rate findings by impact in context (what data, what action, what user population), not by how clever the technique was. Agent findings should be rated by what the credentials allowed.
Model-graded scoring: where an LLM judges outputs, calibrate it against human-labelled samples and spot-check its decisions; an uncalibrated judge silently inflates or hides results.
Measurement after red teaming: turn confirmed failure modes into fixed evaluation datasets so that the rate can be tracked release by release. HarmBench is an example of a published "standardized evaluation framework for automated red teaming."

Regulatory and standards expectations

EU AI Act, Article 55(1)(a): providers of general-purpose AI models with systemic risk must "perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks." Article 55 also requires serious incident reporting to the AI Office and "an adequate level of cybersecurity protection." For high-risk systems, Article 15 covers accuracy, robustness and cybersecurity. See the AI Governance Guide.
NIST AI 600-1 (Generative AI Profile, July 2024): red-teaming appears in suggested actions including MP-5.1-005 ("Conduct adversarial role-playing exercises, GAI red-teaming, or chaos testing"), MS-2.7-007 on resilience, MS-2.8-002 ("Document the instructions given to data annotators or AI red-teamers") and MS-4.2-001 ("adversarial testing at a regular cadence to map and measure GAI risks").
Government evaluators: the UK AI Security Institute, part of the Department for Science, Innovation and Technology, states that it has "evaluated the cyber capabilities of leading open and closed weight AI models", and NIST's Center for AI Standards and Innovation (CAISI) publishes cyber capability assessments of AI models. Their work informs what frontier model providers test for; it does not replace your own application testing.

Running an AI red teaming programme

Inventory and prioritise: list AI systems by exposure (public, internal), data sensitivity and agency (read-only, can act). Agents with write access and public chatbots come first. See the Shadow AI Discovery Guide.
Threat model: fill in the system, actor, TTP, weakness and impact template for each in-scope system and agree the harm taxonomy.
Build the environment: production-like configuration, synthetic data, dedicated test identities, logging of every model call and tool call, and a tested kill switch.
Round 1, open-ended: human testers explore freely; capture everything in a shared log.
Round 2, guided and automated: probe the harm list with automated tools and targeted manual work; measure ASR per objective.
Report and fix: after each round, report top issues, link raw data and preview the next round, as Microsoft recommends.
Retest and gate: turn each confirmed finding into a regression test in CI; a model, prompt or tool change that reopens a fixed failure blocks release or rolls back to the previous configuration.
Repeat on change: new model versions, new tools, new data sources and new user groups all reopen scope.

Keeping testers and systems safe

Tester wellbeing: harm testing exposes people to disturbing content. Microsoft calls for "processes that enable operators to disengage when needed" and "resources to support their mental health." Rotate assignments, cap exposure time and add content warnings to reports.
Sandbox agent tests: evaluations of capable agents have reached real systems. See the Anthropic Claude evaluation incidents, where misconfigured cyber tests let AI models breach four real organisations, and the UK AISI agent testing incident, where agents in a cyber evaluation targeted real people.
Egress and credential control: restrict network egress to an allowlist, give agents short-lived test credentials, and never place real cloud or production keys inside the test environment. The OpenAI and Hugging Face breach describes agents escaping an evaluation sandbox and taking over cloud credentials.
Stop conditions: any unexpected external connection, real data exposure or out-of-scope action halts the test until the cause is understood.

Where identity fits

For agents, the most serious findings are rarely about what the model says; they are about what it does with the access it holds. Every agent under test has a non-human identity: an API key, OAuth token, service principal or workload identity. Red team findings on excessive agency should be traced to the specific scopes and permissions that made them possible, and fixed there, through narrower scopes, per-action authorisation and approval gates, not only with prompt filters. The scenario set for identity abuse (privilege escalation, credential discovery, confused deputy and delegation abuse) is covered in Red Teaming AI Agents for Identity Abuse; authorisation patterns are in the AI Agent Authorisation Guide.

Reporting and remediation

Finding record: system and version, actor, technique, weakness, impact, ASR with trial count, reproduction identifier, severity and recommended fix layer.
Fix in the right layer: model (provider report or fine-tuning), application (system prompt, retrieval permissions, output encoding), guardrails (input and output filters) or identity and architecture (tool removal, scopes, approvals, isolation of untrusted content). Architectural fixes last longest. See the AI Guardrails and AI Firewalls Guide.
Accept residual risk explicitly: some failure rates can only be reduced, not removed; record the accepted ASR and the compensating monitoring.

Detection and measurement

Coverage: percentage of production AI systems red teamed in the last 12 months or since their last major change.
ASR trend per critical objective: before and after mitigation, and across model versions.
Time to remediate: median days from confirmed finding to verified fix, by severity.
Regression rate: fixed failures that reappear in CI after a model, prompt or tool change.
Detection of test activity: share of red team attacks that production monitoring flagged, a direct test of runtime detection.

Framework mapping

Framework	Reference	Relevance
EU AI Act	Art. 55(1)(a); Art. 15	Documented adversarial testing for systemic-risk GPAI models; robustness and cybersecurity of high-risk systems
NIST AI 600-1	MP-5.1-005, MS-2.7-007, MS-2.8-002, MS-4.2-001	Red-teaming, documentation of tester instructions, regular adversarial testing
OWASP Top 10 for LLM Applications 2025	LLM01 to LLM10	Test categories and finding classification
OWASP GenAI Red Teaming Guide	Model, implementation, infrastructure, runtime	Scoping structure for engagements
MITRE ATLAS	Tactics and techniques against AI systems	Common language for TTPs in reports

Common failure patterns

Testing only the model: a clean base model result says nothing about the retrieval permissions, tools and output handling of the product built on it.
Treating a handful of examples as a rate: or the reverse, declaring a system safe because a benchmark passed. Red teaming finds failure modes; measurement sizes them.
Testing on one endpoint and shipping another: different model versions, filters or system prompts in production invalidate the result.
Unscoped agent tests: agents with real credentials or open egress during evaluation, which is how tests have reached real organisations.
Fixing with filters only: blocking the exact prompts the red team used while the agent keeps the over-broad permissions that made the impact possible.
One-off exercises: no regression suite, so the next model update silently reopens fixed failures.

Practitioner checklist

Inventory AI systems and prioritise by exposure, data sensitivity and agency.
Scope each engagement by layer (model, application, agent) using a system, actor, TTP, weakness and impact template.
Agree a written harm taxonomy and rules of engagement, including prohibited actions and disclosure routes.
Run agent tests in a sandbox with test identities, egress allowlists, full logging and a tested kill switch.
Start with open-ended human testing, then guided and automated rounds.
Record every attempt with versions and identifiers, and report ASR with trial counts.
Fix in the most durable layer, especially identity and permissions for agents.
Convert findings into CI regression tests that gate releases.
Provide wellbeing support and exposure limits for testers.
Map evidence to EU AI Act Article 55 and NIST AI 600-1 where they apply.

AI Red Teaming Guide