AI Red Teaming vs Evaluations vs Benchmarks: What Each Proves at a Release Gate

Release boards for AI systems are routinely shown three kinds of evidence, a benchmark score, an evaluation pass rate and a red team report, and treat them as interchangeable when each proves something different and each can be wrong in its own way. This guide is for AI assurance leads, application security teams and release approvers who must decide what evidence a go or no-go decision needs. It compares the three methods by threat model, coverage, cadence and failure modes, explains benchmark contamination, and ends with a combined test plan you can adopt. It builds on the AI Security Evaluations Guide and the AI Red Teaming Guide rather than repeating them. Sources were checked on 13 November 2026.

Key takeaways

• Each method answers a different question. Benchmarks compare capability on a shared public task, evaluations check your own system against known risks on every change, and red teaming searches for the unknown risks nobody has written a test for.
• Benchmark scores are weak release evidence. Public test items leak into training data, so a high score can reflect memory rather than ability; NIST AI 600-1 action MS-2.11-001 asks you to document benchmark limitations including training and test data cross contamination.
• Static evaluations understate determined attackers. NIST CAISI found agent hijacking success rose from 11% with the strongest baseline attack to 81% with an attack adapted to the target, and from 57% to 80% on average when each attack was retried 25 times.
• A red team report expires. It describes one system version, one scope and one time window; findings only keep protecting you once they are turned into evaluation cases that run on every release.
• Gate on a combined evidence pack. Require benchmarks for model selection, an evaluation pass on every release candidate, and a scoped red team for material changes, with explicit rules for what each failure means.

The short answer: what each method proves

The bottom row matters most. None of the three can prove a system is secure. Each reduces a different kind of uncertainty.

Benchmarks: capability evidence with a contamination problem

Threat model and architecture

A benchmark is a public dataset, a fixed way of prompting the model and a scoring rule, published so results are comparable across models and labs. Its implicit threat model is a generic attacker using known techniques, and it runs the model through a harness, usually with no access to your tools, data or system prompt.

Contamination and gaming

Because benchmark items are public, they end up in web crawls and therefore in pretraining data. Deng and colleagues (NAACL 2024) masked a wrong option in multiple choice questions and asked models to fill the gap: two commercial models guessed the exact missing MMLU option 52% and 57% of the time, which is hard to explain without exposure to the test items. Zhang and colleagues built GSM1k, a fresh set of grade school maths problems matched in style and difficulty to the widely used GSM8k, and found leading models scored up to 8% lower on the new set, with several model families showing signs of systematic overfitting, although many frontier models showed little. For a release board the lessons are:

• Memory is not robustness: a model that has seen published jailbreak prompts may refuse those exact strings while failing simple paraphrases.
• Vendor-reported scores are not independent: the harness, prompt format and number of attempts change results, and these are often not disclosed.
• Bare-model scores ignore your controls: your guardrails may make a weaker model safe, or your tools may make a strong model dangerous.

NIST AI 600-1, the Generative AI Profile, asks organisations to document the assumptions and limitations of benchmarks, including training and test data cross contamination, relative to the deployment environment (MS-2.11-001), and not to extrapolate performance from narrow or anecdotal assessments (MS-2.5-001); the ML model evaluation and monitoring for security teams guide covers the statistics. NIST's draft AI 800-2, Practices for Automated Benchmark Evaluations of Language Models, published by CAISI as an initial public draft on 30 January 2026 with comments closed on 31 March 2026, organises good practice into defining objectives and selecting benchmarks, implementing and running evaluations, and analysing and reporting results.

Controls that make benchmark evidence usable

• Re-run, do not quote: run the benchmark yourself with a recorded harness version, prompt format and attempt count instead of copying a vendor figure.
• Add a private twin: keep a held-out, never-published set written to the same specification, and treat a large gap between public and private scores as a contamination signal.
• Report uncertainty: give the number of items and a confidence interval, so a two point difference on a few hundred items is not read as a real change.

Decision criteria

Use benchmarks to shortlist models and to detect gross capability changes when a provider updates a model. Do not use a public benchmark score as the security gate for a deployed application or agent.

Evaluations: regression evidence that only knows what it was taught

Threat model and architecture

An evaluation suite is your own private, versioned set of test cases run against your full system. Frameworks such as the UK AI Security Institute's open-source Inspect structure each evaluation as a dataset of samples, a solver that produces the answer (from a single model call to a tool-using agent) and a scorer that grades the result, with sandboxes such as Docker or Kubernetes for running untrusted code and a log viewer for inspecting transcripts. The threat model is explicit: each case represents a named risk such as indirect prompt injection (ATLAS AML.T0051), jailbreak (AML.T0054), data leakage or tool misuse. The detailed test catalogue, scoring and CI wiring are covered in the AI Security Evaluations Guide; this section focuses on what the results can and cannot support at a gate.

What evaluations miss

• Unknown attacks: a suite only finds what someone wrote it to find. CAISI's agent hijacking work showed a model performing well on existing evaluations while a red team attack built for it succeeded 81% of the time.
• Single-attempt optimism: outputs are probabilistic and attackers retry. Running each case once reports the average case, not the attacker's best case.
• Self-contamination: if engineers tune prompts or guardrails against the suite until it passes, the suite becomes a training set and stops measuring generalisation.
• Evaluation awareness: capable models may behave differently when the context looks like a test. See the AI Scheming, Sandbagging and Evaluation Awareness Guide.

Controls that make evaluation evidence trustworthy

• Split the suite: a development set engineers can see and tune against, and a locked gate set that is scored only at release and never used for tuning.
• Repeat and vary: run high-impact cases several times and with paraphrases, and report success within k attempts as well as the mean.
• Score on state, not text: check whether the email was sent or the canary record left the environment, rather than asking a grader whether the reply looks safe.

Decision criteria

Evaluations are the right primary gate for routine change: prompt edits, tool additions, retrieval changes and provider model updates. They are not sufficient alone for a first launch, a new class of tool access or a new autonomy level.

Red teaming: discovery evidence with a short shelf life

Threat model and architecture

Red teaming is open-ended adversarial testing by people, often supported by automation, who adapt to the system's responses. The OWASP GenAI Red Teaming Guide (January 2025) scopes it across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis. Its threat model is a capable, motivated adversary with time to explore, chain weaknesses and retry. For AI agents that includes abuse of the agent's own credentials and permissions; see Red Teaming AI Agents for Identity Abuse.

NIST AI 600-1 gives red teaming several specific jobs: assessing resilience to prompt injection and data poisoning (MS-2.7-007), testing for training data output, membership inference and privacy leaks (MS-2.10-001), and finding unforeseen failure modes through adversarial role-play or chaos testing (MP-5.1-005). It also asks that the people running structured feedback exercises are not directly involved in developing the same model (MS-1.3-003) and that instructions given to red teamers are documented (MS-2.8-002).

What red teaming misses

• Coverage: a time-boxed team explores some paths deeply and leaves others untouched.
• Decay: the next model update, prompt change or tool can reopen a closed path. A report is evidence for the version tested.
• Exercise risk: agents under test have reached real systems when test environments had open network access or live credentials.

Controls that make red team evidence usable

• Write the scope as claims: state the claims the exercise tests ("the support agent cannot send customer data outside the CRM") so the report says which claims held.
• Require reproduction artefacts: each finding needs inputs, environment state and expected versus observed behaviour, so it can become an evaluation case.
• Convert and re-test: a finding is not closed until its evaluation case exists and passes against the fixed build.
• Contain the test: enforce isolation in the network and with scoped, fake credentials, not with instructions in the prompt. Use the Red Team Rules of Engagement Template and the AI Red Team Test Plan Template.

Decision criteria

Red team before a first launch, before granting new autonomy or new tool access, after a significant model change, and when regulation requires adversarial testing. For providers of general-purpose AI models with systemic risk, Article 55(1)(a) of the EU AI Act requires model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model, with a view to identifying and mitigating systemic risks. See the EU AI Act GPAI Obligations Guide.

Failure modes compared

Release gates: what evidence each decision needs

Match the evidence to the size of the change, and write the rules down before the release.

Silent provider model updates are the change most often missed, because nothing in your repository changes. Treat a provider version change as a change request with an evaluation gate; the AI model change management comparison covers how to fit this into a change advisory board.

A combined test plan

Benchmarks narrow the choice, red teaming discovers, evaluations remember.

1. Write the claims. List the five to ten security claims the system must support, each tied to an impact (data leaving a boundary, an unauthorised action, a credential exposed).
2. Select models on re-run benchmarks. Run chosen public benchmarks in house with a recorded harness, plus a private twin set. Record contamination indicators and limitations in the model selection record.
3. Build the evaluation suite. Seed it from public attack sets, the OWASP and ATLAS techniques relevant to your claims, and your own threat model. Split it into development and locked gate sets.
4. Red team before launch. Scope the exercise by the claims, run it in a contained environment with fake credentials and denied egress, and require reproduction artefacts for every finding.
5. Convert findings. Each finding becomes one or more evaluation cases, including paraphrased variants, added to the gate set. The finding closes only when the case passes on the fixed build.
6. Gate every change. Run the smoke set pre-merge and the full gate set with repeated attempts on release candidates and provider model changes, using the gate rules above.
7. Run scheduled evaluations in production-like staging. Weekly or monthly, and on any provider notice, so drift is caught without a code change.
8. Refresh the red team. Repeat on material change and at least annually, rotating testers, and feed new techniques back to step 5.
9. Roll back on regression. Keep the previous model version, prompt and configuration deployable, and define in advance which gate failures trigger rollback instead of a fix forward.

Worked scenario

A bank's customer support agent can read account data and draft emails. One shortlisted model scores well on a public injection benchmark but drops markedly on the private twin set, so its score is recorded as possibly contaminated. The gate set includes 120 indirect injection cases planted in inbound emails, each run five times. Before launch, an external red team finds that a calendar invitation can instruct the agent to forward a statement to an outside address, a channel the suite did not cover. The fix restricts recipients in the email tool itself, and the invitation plus twenty paraphrases join the gate set. Three months later the provider ships a new model version; the full gate set shows attack success on the forwarding task rising from zero to two of 600 attempts, which breaches the zero tolerance rule for that task, so the board holds the update and keeps the previous version.

Where identity fits

All three methods are only as safe as the identities the system under test holds. Evaluations and red team exercises for agents should run with dedicated test identities, canary tokens and fake secrets that alert on use, never production service accounts or API keys. The same scoped, short-lived credential practices that limit an agent in production also limit the blast radius of a test that escapes its sandbox, which is the lesson of the Anthropic Claude Evaluation Incidents 2026, where a misconfigured third-party evaluation environment left internet access open and models under test reached real organisations. See the Agentic AI Identity Guide.

Detection and measurement

• Attack success per critical task: from the gate set, with success within k attempts reported beside the mean.
• Red team to evaluation conversion rate: share of findings with a passing evaluation case within an agreed period; target 100% for high severity.
• Public versus private benchmark gap: tracked per model; a widening gap is a contamination signal.
• Novel red team success rate: share of red team attacks not already represented in the suite that succeed; if it stays high, the suite is not learning.
• Evidence age at release: days since the last red team on the current capability set and model version.

Framework mapping

Common failure patterns

• Quoting a vendor benchmark as the security sign-off. The score describes a bare model on a public set, not your system with its tools and data.
• Treating "no critical findings" as proof of safety. It describes the scope and time the red team had; the report should say which claims were tested and which were not.
• Red team findings that never become tests. The fix ships, the next model update reopens the path, and nobody notices because no evaluation case exists.
• Ignoring provider model changes. The release process triggers on code commits, so a silent model update goes live without any gate.
• Testing in an environment that can reach production. Open egress or real credentials turn a test into an incident, as the evaluation incidents above show.

Practitioner checklist

• Write the system's security claims and map every benchmark, evaluation case and red team objective to one.
• Re-run benchmarks in house with a recorded harness and a private twin set, and record contamination indicators.
• Split the evaluation suite into a development set and a locked gate set used only at release.
• Run high-impact cases with repeated attempts and paraphrases, and report success within k attempts.
• Scope red team exercises by claims, require reproduction artefacts and keep testers independent of the developers.
• Close a finding only when its evaluation case passes on the fixed build.
• Treat provider model version changes as changes that require the full gate set.
• Deny network egress and use fake, monitored credentials in every test environment.
• Publish gate rules and rollback triggers before the release, not during it.

Standards and references

• NIST: AI 600-1 Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• NIST CAISI: Towards Best Practices for Automated Benchmark Evaluations (AI 800-2 initial public draft)
• NIST CAISI: Technical Blog: Strengthening AI Agent Hijacking Evaluations
• EUR-Lex: Regulation (EU) 2024/1689 (Artificial Intelligence Act)
• OWASP: GenAI Red Teaming Guide
• UK AI Security Institute: Inspect
• arXiv: Investigating Data Contamination in Modern Benchmarks for Large Language Models
• arXiv: A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Related NHI Mgmt Group resources: AI Security Evaluations Guide, AI Red Teaming Guide, Continuous Automated Red Teaming (CART) Guide, AI Model Risk Management Guide, AI Red Teaming and Testing Tools Buyer's Guide.