How should security teams red team a foundation model before deploying it into production?

Security teams should test the model with a structured red team that targets jailbreaks, toxic output, disinformation, and policy bypasses before any production use. The goal is to expose failure modes under adversarial prompting, then tighten system prompts, guardrails, and monitoring. A one-time evaluation is not enough. Models need continuous adversarial testing because attack techniques evolve and safety gaps can reappear after updates.

What a Foundation-Model Red Team Is Trying to Prove

A foundation-model red team is not a general assurance exercise and it is not the same as ordinary functional testing. Its purpose is to find ways the model can be induced to behave unsafely, ignore policy, reveal restricted content, or produce unreliable outputs when an adversary actively tries to steer it. For production readiness, that means testing the model’s behaviour under hostile prompting, prompt chaining, role-play, tool abuse, and edge cases that stress guardrails before real users do.

That distinction matters because foundation models fail in ways traditional software tests often miss. A model can appear safe in benchmark-style evaluation and still be brittle under novelty, ambiguity, or multi-turn manipulation. Red teams therefore need to examine both direct jailbreak attempts and the broader conditions that let policy bypass emerge after deployment, especially when the model is wrapped in applications, tools, or orchestration layers. NIST’s NIST AI 600-1 Generative AI Profile is useful here because it treats generative AI risk as a lifecycle governance issue, not a one-off test event.

In practice, many security teams discover the most serious failures only after testers start combining benign-looking prompts across multiple turns rather than through single-shot jailbreak attempts.

How Security Teams Should Structure the Test

A useful red team starts with scope. Security teams should define exactly what the model is allowed to do, which content or actions are prohibited, which tools it can call, and which downstream systems it can influence. That scope becomes the baseline for attack cases. If the model will summarize documents, answer user questions, draft text, or trigger workflows, each of those behaviours needs a separate test path because safety often degrades when the model is asked to act, not just talk.

The test plan should then cover a few distinct classes of abuse. First, prompt-injection and jailbreak attempts should probe whether the model can be coerced into ignoring system instructions. Second, policy-bypass tests should look for indirect phrasing, translation tricks, role-play, and multi-turn escalation. Third, output-safety tests should measure toxic content, harmful advice, hallucinated confidence, and disallowed persuasion. Fourth, if the model has tool access, the red team should test whether the model can be manipulated into invoking the wrong tool, leaking data, or amplifying an unsafe instruction into an external action.

Good practice is to test across the whole application chain, not only the base model. The wrapper, retrieval layer, tool permissions, and logging all affect the real security posture. A model that behaves acceptably in isolation may fail once retrieval-augmented context, user-provided files, or agentic actions are introduced. That is why results should be scored by scenario, not only by prompt. Teams need to know whether the failure is a harmless wording issue, a policy breach, or a path to operational harm.

Test single-turn and multi-turn manipulation separately.
Include hostile prompts, ambiguous prompts, and benign prompts that turn malicious later.
Record whether the failure is content, policy, tool use, or data exposure.
Retest after model updates, prompt changes, and guardrail changes.

The guidance breaks down when the evaluation only checks static prompts and ignores the surrounding application, because that misses the paths most likely to fail in production.

Where Red Teaming Usually Breaks Down

Tighter safety controls often reduce model usefulness, so teams have to balance blocking harmful output against preserving legitimate capability. That tradeoff becomes sharper when the model is deployed for customer-facing or internal productivity use, because overly restrictive guardrails can drive users to work around the system instead of through it.

One common edge case is the difference between direct harmful output and indirect harmful enablement. Some teams focus only on toxic text or obviously dangerous advice, but a model can still create risk by helping a user refine disinformation, automate abuse, or discover a better evasion strategy. Another edge case is update drift: a model that passed a red team last month may behave differently after a new checkpoint, retrieval source, or prompt template is introduced. Guidance on the exact red-team cadence is still not fully standardised across the industry, but continuous testing is the safer operational assumption.

Another overlooked issue is false confidence from isolated success cases. A few blocked jailbreaks do not prove resilience if the same weakness reappears under paraphrase, language switching, or longer context windows. For that reason, teams should treat the red team as a regression discipline, not a launch gate that ends once the first report is delivered.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATLAS address the attack surface, NIST AI 600-1 and NIST AI RMF set the technical controls, and ISO/IEC 42001:2023 define the regulatory obligations.

Framework	Control / Reference	Relevance
NIST AI 600-1	GOV-1 — Map, Measure, and Manage Risks	Generative AI red teaming directly supports lifecycle risk measurement.
	MAP-2 — AI Context and Intended Use	Scope definition depends on intended use and deployment context.
Recommendation — Use GOV-1 to test model behavior against defined misuse and safety risks. Define intended use and prohibited use cases before testing.
ISO/IEC 42001:2023	6.1 — Actions to Address Risks and Opportunities	Red teaming is a core AI risk treatment and validation activity.
Recommendation — Embed red-team findings into AI risk treatment and approval decisions.
NIST AI RMF	MEASURE — Measure	Red teaming measures harmful behavior under adversarial prompts.
Recommendation — Measure model failure modes under hostile prompting before release.
MITRE ATLAS	ATLAS T0001 — Elicit model behavior	Jailbreak and prompt attacks elicit unsafe model behavior.
Recommendation — Map jailbreak scenarios to elicitation techniques and test for them explicitly.

Practitioner Guidance

What to prioritise: Start with the paths that can change real-world impact, especially tool use, retrieval context, and any output that can trigger external action. Those are the cases where a model failure becomes an operational failure rather than just a bad answer.

What to verify: Confirm that the red team tests the wrapped application, not only the base model. If the guardrails do not cover multi-turn manipulation, retrieval injection, and post-generation actions, the assessment is incomplete even if the model looks strong in isolation.

What practitioners underestimate: The most important question is often not whether the model can be tricked once, but whether the same weakness returns after prompt, data, or model updates. Production readiness depends on whether the organisation can detect drift and rerun the same abuse cases consistently.

Practitioner takeaway: Treat foundation-model red teaming as a repeatable control over the full application stack, because the highest-risk failures usually appear when a model is combined with tools, context, and real users.