LLM red teaming tools still need governance beyond discovery

At a glance

What this is: This is a practitioner-focused comparison of LLM red teaming tools, with Braintrust arguing that the real value comes after discovery when confirmed failures are turned into versioned evaluations and CI checks.

Why it matters: It matters because AI security programmes need evidence that survives release cycles, and IAM, NHI, and agentic AI teams must govern not just model behaviour but the identities, prompts, tools, and workflows those systems touch.

👉 Read Braintrust's comparison of the best LLM red teaming tools for 2026

Context

LLM red teaming is a control problem as much as a testing problem. Discovery scans can surface attacks quickly, but they do not by themselves answer whether a failure is real, whether it recurs after a model or prompt change, or whether it should block release. For identity and access teams, the issue becomes sharper when LLMs act through tools, APIs, and delegated permissions because the system’s runtime behaviour can cross into NHI governance.

The article is useful because it separates attack generation from ongoing evaluation. That distinction matters in agentic AI environments where prompt injection, excessive agency, and tool misuse can create access risk that looks operational at first and becomes governance debt later. The starting point described here is typical of mature teams: they need both discovery and durable control, not a single red-team run.

Key questions

Q: How should teams turn LLM red team findings into lasting controls?

A: Teams should preserve confirmed failures as versioned evaluation cases with a clear scorer, owner, and expected safe behaviour. That lets the same issue run again after model, prompt, retrieval, tool, or policy changes. The goal is not to keep generating attacks forever, but to convert proven weaknesses into repeatable release checks.

Q: Why do endpoint agentic AI tools create more governance risk than chat-only GenAI?

A: Endpoint agentic AI can act inside a user’s session, move data, and trigger downstream actions, which expands the effective privilege boundary. Chat-only tools may still create data risk, but agents can combine access and action in ways that are harder to see and easier to over-scope. That makes workflow control and telemetry more important than simple application approval.

Q: What do security teams get wrong about LLM red teaming?

A: They often stop at discovery and treat a successful attack as a report instead of a control input. Without durable evaluation cases, the same failure can reappear after a harmless-looking release change. The right operating model treats the red-team corpus as living evidence for engineering and governance.

Q: How can organisations know whether LLM red team testing is actually working?

A: Look for repeatability, traceability, and release impact. If a known failure reruns after change, still fails, and can block promotion when risk is unresolved, the programme is working. If findings vanish into spreadsheets or one-off tickets, the test is producing noise rather than governance value.

Technical breakdown

Attack generation versus persistent evaluation in LLM security

LLM red teaming tools usually fall into two layers. The first layer generates adversarial prompts, multi-turn conversations, or tool-use sequences to provoke unsafe behaviour. The second layer preserves confirmed failures as versioned cases with expected safe outcomes, so the same issue can be retested after model, prompt, retrieval, or policy changes. That separation is important because a finding is only operationally useful when it can be reproduced, scored, and rerun as the application evolves. In agentic systems, the same logic applies to tool calls and delegated actions, not just text output.

Practical implication: Separate exploratory scanning from regression evaluation so confirmed failures become release checks rather than one-off findings.

Why agent and tool coverage changes the red team target

Modern red teaming is no longer limited to a raw model endpoint. Applications may include retrieval pipelines, browser automation, external tools, memory, or policy layers that expand the attack surface beyond the LLM itself. A useful tool therefore needs to exercise the whole workflow, because risk often emerges when the model can select actions, call tools, or carry state across turns. This is where identity intersects with AI security: the permissions attached to agents, service accounts, and API keys determine how far a successful prompt can travel. Without that view, teams test language behaviour while missing authorization behaviour.

Practical implication: Test the full agentic workflow, including tool permissions and delegated credentials, not only the model response.

Scoring matters because not every jailbreak is an operational risk

A red-team scan produces noise unless the scoring logic reflects the behaviour you actually care about. Security teams need checks for unauthorized actions, sensitive-data exposure, policy bypass, and unsafe completion patterns, not just generic model refusals. The strongest workflows store the attack input, metadata, expected safe behaviour, and a scorer that can confirm whether the same failure still exists. That creates a repeatable control point across application changes. In practice, scoring becomes the bridge between security testing and governance because it defines what counts as an unacceptable outcome.

Practical implication: Define scorers around unauthorized actions and data exposure so tests map to governance decisions, not vanity metrics.

Threat narrative

Attacker objective: The objective is to convert model weakness into operational abuse of tools, data, or delegated access within the application workflow.

1. Entry occurs when an attacker uses crafted prompts, multi-turn conversations, or poisoned context to probe an LLM-backed application for unsafe behaviour.
2. Escalation happens when the model is induced to reveal data, bypass policy, or invoke tools and delegated permissions beyond intended scope.
3. Impact follows when the agent or model action reaches connected systems, exposing sensitive information, triggering unauthorized actions, or creating repeatable abuse paths.

NHI Mgmt Group analysis

Braintrust’s real significance is that it treats confirmed AI failures as governance objects, not just test artefacts. That matters because a red-team hit only becomes useful when it can be versioned, scored, and rerun after change. In practice, this moves AI security from discovery theatre to control enforcement, which aligns more closely with how mature IAM and PAM programmes manage exceptions and lifecycle state.

LLM red teaming is becoming an identity problem because agent behaviour is defined by access, not only language. Once an AI system can call tools, retrieve data, or act on behalf of a user, the security question shifts to delegated authority, credential scope, and runtime boundaries. That is why OWASP NHI and agentic AI governance concepts belong in the same conversation as model evaluation.

Versioned evaluation corpora create the missing control plane for AI governance debt. Teams can often discover a failure quickly, but they struggle to keep that failure visible after the next sprint, model swap, or policy adjustment. Durable cases, especially when linked to release gates, are the practical answer because they turn a known weakness into an enforceable engineering obligation.

LLM red teaming tools are fragmenting along the same line as identity tooling did: discovery on one side, lifecycle governance on the other. Garak, PyRIT, and Promptfoo help surface weaknesses, while Braintrust represents the category of control that retains evidence and operationalises it. Practitioners should expect AI security programmes to mature only when they can connect tests to ownership, reruns, and change management.

Named concept: evaluation persistence. This is the point at which a confirmed LLM failure survives beyond the original test run and becomes part of the security programme’s institutional memory. It is the difference between a red-team finding and a governable risk that can block release or trigger review.

What this signals

The next step for AI security programmes is to stop treating red team output as a testing artefact and start treating it as a governed inventory of known failure modes. That inventory should connect to release management, ownership, and access scope, especially where agents can invoke tools or inherit delegated credentials.

Evaluation persistence: confirmed AI failures need to survive model updates, prompt changes, and policy edits if they are to influence security decisions. Teams that cannot preserve those cases are likely to rediscover the same weaknesses repeatedly, particularly in agentic workflows that cross identity boundaries. For a governance baseline, align the testing lifecycle with the OWASP Agentic AI Top 10 and the NIST AI 600-1 Generative AI Profile.

The operational signal is whether your programme can answer three questions quickly: which failures are real, which ones recur, and which ones block change. If those answers are not visible, the AI estate is being tested, but not governed.

For practitioners

• Build a two-stage red-team workflow Use one tool for attack generation and a separate evaluation layer for confirmed failures, then require versioned cases before anything becomes a release gate.
• Tie AI tests to tool and permission scope Map every red-team case to the model, retrieval source, tool call, and delegated credential it exercised so access scope stays visible during review.
• Create scorers for unsafe actions, not just unsafe text Score unauthorized actions, data exposure, and policy bypass so the test reflects operational risk instead of refusal quality alone.
• Rerun known-risk cases after relevant change Schedule reruns when prompts, models, retrieval sources, policies, or tool permissions change, and treat failures as regression evidence.
• Preserve attack transcripts with ownership metadata Store the original prompt or transcript, severity, affected feature, and owner so security and engineering can trace why the case matters.

Key takeaways

• LLM red teaming is most effective when discovery and regression testing are separated into distinct controls.
• Agentic AI raises the stakes because tool use and delegated access turn model behaviour into an identity and authorization problem.
• Durable evaluation cases are the practical mechanism that turns confirmed failures into security governance, not just test output.

Key terms

• Evaluation Persistence: Evaluation persistence is the practice of preserving confirmed AI failures as durable test cases that survive application change. It turns one-off red-team discoveries into repeatable controls, making it possible to rerun the same risk after updates to prompts, models, retrieval, policies, or tool permissions.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Red-Team Corpus: A red-team corpus is the curated set of adversarial prompts, transcripts, and failure cases used to test an AI system over time. In mature programmes, it includes metadata, expected safe behaviour, ownership, and rerun conditions so the corpus supports governance as well as testing.

What's in the full article

Braintrust's full article covers the operational detail this post intentionally leaves for the source:

• Side-by-side product criteria for Garak, Promptfoo, PyRIT, DeepTeam, Mindgard, and Braintrust
• Pricing notes, licensing details, and deployment model differences that matter during tool selection
• Workflow guidance for retaining confirmed attacks as versioned evaluations and CI checks
• Practical examples of when to use discovery scanning versus regression testing

👉 The full Braintrust article covers tool selection criteria, workflow trade-offs, and where each option fits in production evaluation

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management in the context of real-world control design. It is suited to practitioners who need to connect identity governance with emerging AI-operated workflows.