AI red teaming exposes prompt failure modes in agent security

At a glance

What this is: This walkthrough shows how AI red teaming, tracing, evaluations, and prompt optimisation can be combined into a repeatable safety-improvement loop for LLM applications.

Why it matters: It matters because AI security teams need evidence-based controls for adversarial prompts, not one-off testing, and identity-led programmes should watch for where agent access and safety governance intersect.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing credentials.

👉 Read Arize's walkthrough of AI red teaming and prompt optimisation in Microsoft Foundry

Context

AI red teaming for LLMs is a controlled way to test how an AI system behaves under adversarial prompting, safety abuse, and jailbreak attempts. The governance gap is not whether these attacks exist, but whether teams can observe, evaluate, and respond to them consistently across the model lifecycle.

In this article, Arize frames safety testing as a repeatable loop: probe the model, capture traces, score regressions, annotate failures, improve prompts, and retest. That aligns closely with the broader AI governance challenge, where model behaviour, access to data, and the identity of the system acting on behalf of users all need clear controls.

For identity and security teams, the interesting point is not the prompt optimiser itself but the operational pattern around it. The starting position is increasingly typical for AI teams that have testing in place but lack mature governance over failure handling and continuous assurance.

Key questions

Q: How should organisations red team AI models before production?

A: Use both automated and manual testing so you cover scale and adversarial nuance. Focus on abuse classes such as jailbreaks, misinformation, bias, harmful content, and tool misuse, then tie each finding to a mitigation owner and a retest. Red teaming only matters when it changes release decisions and closes the loop on risk.

Q: Why do AI red team failures need to be tracked as governance evidence?

A: Because a failed prompt often reveals a repeatable weakness rather than an isolated error. Tracking it as evidence lets teams compare versions, identify regressions, and show whether remediation actually reduced risk. That matters for compliance, auditability, and any programme that needs defensible AI safety assurance.

Q: What do organisations get wrong when they automate prompt optimisation?

A: They often confuse suggestion generation with safe deployment. Automated optimisation can help identify better instructions, but it should not bypass review, dataset curation, or rollback planning. If the human checkpoint disappears, the organisation may simply automate the wrong behaviour faster.

Q: How do you know whether AI red teaming is actually improving governance?

A: Look for repeatable reductions in high-severity findings, clearer ownership of model and tool permissions, and evidence that tests are blocking risky releases. If findings are interesting but do not change access scope, secrets handling, or deployment decisions, the programme is producing noise rather than control.

Technical breakdown

How AI red teaming finds jailbreak and safety failures

AI red teaming is adversarial testing aimed at eliciting unsafe model behaviour rather than exploiting code or infrastructure flaws. The attacker role is simulated through prompts that use role-play, hypotheticals, and gradual escalation to bypass refusals and policy boundaries. In this workflow, the model is not compromised in a traditional sense. Instead, its instruction hierarchy and safety filters are stressed until the boundary between benign assistance and harmful output becomes visible. That makes red teaming useful for measuring how stable a model remains when the prompt surface is manipulated repeatedly.

Practical implication: teams need red-team coverage for prompt abuse patterns, not just static policy checks.

Why traces and evaluations matter after the attack attempt

Red teaming only becomes operationally useful when every attack attempt is captured as trace data and scored with consistent evaluations. Traces preserve the prompt, model response, and surrounding context, while evaluations convert subjective failure into labelled outcomes such as pass or fail. This matters because safety regressions are often pattern-based, not single-event failures. A model may refuse obvious abuse but fail on subtle variants. Without evaluation labels, teams cannot compare prompt versions, track regressions, or separate isolated anomalies from repeatable weaknesses.

Practical implication: instrument red-team runs with evaluation labels so failures become measurable control signals.

How prompt optimisation turns failed probes into a control loop

Prompt optimisation uses failed examples as training signals for the system prompt rather than retraining the base model. In effect, the model’s operating instructions are iterated against curated regressions, with human review used to create golden datasets where needed. This is not autonomous security hardening. It is a governance loop that couples machine-generated feedback with human approval before deployment. The important architectural point is that the control plane sits around the prompt, not inside the model weights, so the workflow can improve guardrails without changing the underlying model provider.

Practical implication: treat prompt optimisation as controlled governance over instructions, with human review before any production change.

Threat narrative

Attacker objective: The objective is to induce unsafe or policy-violating model behaviour and identify which prompt patterns break the system's defences.

1. Entry begins with adversarial prompts that target the model through ordinary chat or API inputs, using role-play, hypothetical framing, and escalation patterns to bypass safety boundaries.
2. Escalation occurs when the model produces disallowed content or misses a refusal path, allowing the tester to map which prompt variants defeat the guardrails and where the system is inconsistent.
3. Impact is the exposure of repeatable safety regressions that can be turned into stronger prompts and better controls, reducing the chance of harmful outputs in production.

NHI Mgmt Group analysis

AI red teaming is becoming a governance control, not just a testing exercise. The article shows that the value is not in generating adversarial prompts alone, but in converting failures into auditable improvements. That is an AI security and model governance pattern, but it also intersects with identity because the behaviour of an AI system acting on behalf of users can create access, data, and privilege risk. Teams should treat red teaming output as control evidence, not as a one-off safety report.

Prompt optimisation creates governance debt if human review is skipped. The workflow described here works because failed probes become labelled regressions and then curated datasets. If organisations automate the next step without review, they risk encoding the wrong safety assumptions into production prompts. The practical lesson is that AI safety controls need approval paths, provenance, and rollback, especially where the model can influence sensitive workflows.

Traceability is the named concept this article surfaces. Traceability gap: without trace capture, evaluation labels, and linked failure cases, teams cannot prove why a model refused, failed, or changed after remediation. That matters for AI governance, incident response, and compliance evidence. In practice, the post highlights that observability is the bridge between red teaming and accountable remediation.

This workflow validates continuous assurance over periodic benchmarking. Static safety tests age quickly because attackers adapt their phrasing, tone, and escalation paths. A loop that retests, scores, and updates prompts is closer to operational assurance than compliance theatre. For practitioners, the field is moving toward living controls that must be revisited as models, prompts, and policies change.

AI security programmes will increasingly need to coordinate with identity and access governance. As AI systems gain access to tools, datasets, and agentic workflows, the real question is who or what is allowed to act, and under what conditions. That brings model governance into the same conversation as entitlement review, privileged access, and workload identity. Practitioners should align safety testing with access control review rather than treat them as separate programmes.

What this signals

Traceability gap: AI teams that cannot connect attack prompts to evaluation outcomes will struggle to prove whether safety controls are improving or merely changing the failure shape. That pushes programmes toward evidence-led remediation, with observability and review trails becoming core parts of AI governance rather than optional telemetry.

Where AI systems can use tools or access data, prompt safety and identity governance start to overlap. Teams should watch for delegated actions that outgrow their original purpose, and align red-team findings with access reviews, tool permissions, and workload identity controls before agentic behaviour expands further.

The practical signal is that continuous assurance will replace periodic model reviews for higher-risk deployments. Organisations that can turn regressions into structured datasets and approved prompt changes will move faster without losing control, while those that cannot will keep discovering the same failures under different wording.

For practitioners

• Instrument red-team runs with end-to-end tracing Capture the original prompt, model response, safety label, and remediation path so each failure can be analysed later as evidence rather than anecdote.
• Build a curated regression dataset from failed prompts Separate obvious refusals from subtle failures, then curate a golden dataset that reflects the specific patterns your red-team probes exposed.
• Require human approval before prompt changes reach production Use prompt optimisation to propose candidate updates, but keep human review in the loop for safety, policy, and rollback decisions before deployment.
• Connect model safety evidence to access governance Where the model uses tools or data sources, review whether the same traces reveal inappropriate access paths, over-broad permissions, or weak delegation boundaries.

Key takeaways

• AI red teaming becomes materially useful only when failures are traced, labelled, and fed back into the control loop.
• The article shows why governance around prompt changes matters as much as the red-team scan itself.
• As AI systems gain tool access, red-team evidence should inform both model safety and identity governance decisions.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Return Optimisation: Return optimisation is the redesign of return policy and workflow so merchants reduce cost, abuse, and operational drag without making legitimate returns painful. It combines customer experience, fraud prevention, and process efficiency into one control problem rather than treating returns as a pure logistics function.
• Regression Dataset: A regression dataset is a curated set of past failures and representative edge cases used to check whether a proposed fix actually improves a system without reintroducing the original problem. In AI operations, it preserves the evidence that exposed the defect.
• Identity Traceability: Identity traceability is the ability to link each action back to a specific identity, authorisation path, and time window. It is essential when humans, service accounts, and AI agents all operate in the same environment and auditors need a defensible record.

What's in the full article

Arize's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step setup for the Microsoft Foundry red teaming agent and Arize tracing instrumentation.
• Example notebook flow for converting failed red-team runs into a curated regression dataset.
• Prompt Hub optimisation workflow showing how successive prompt versions are generated and compared.
• Before-and-after scoring results that quantify the effect of the optimisation loop.

👉 Arize's full post covers the scan workflow, dataset creation, and prompt iteration details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It helps security and identity teams connect AI control gaps to the governance models their broader programme depends on.