AI evaluation gaps widen when CI/CD, tracing, and red teaming split

At a glance

What this is: Braintrust's comparison shows that AI evaluation is splitting into two operating models: local red teaming in YAML and production-linked scoring tied to traces, release gates, and human review.

Why it matters: For IAM, NHI, and AI governance teams, the issue is whether evaluation controls are only developer-local or also enforced where production behavior, delegated actions, and release decisions actually happen.

Context

AI evaluation is the control layer that checks whether model output, agent behaviour, and safety tests match what the business expects before and after release. When evaluation stays in a local test harness, it can miss production traces, real user inputs, and release-time enforcement, which creates a governance gap for teams shipping AI into live workflows.

The article frames a common split in AI operations: one tool favours developer-led red teaming and repository-based tests, while the other ties traces, scoring, CI/CD gates, and human review into a continuous workflow. For IAM and agentic AI governance, the important question is whether decision-making over AI behaviour extends into production, where access, prompts, and tool use can have real operational impact.

That split is typical of teams whose AI programmes matured in development first and only later had to prove control in production. The stronger the production dependency, the less adequate a purely local evaluation model becomes.

Key questions

Q: How should teams evaluate AI systems when the model uses tools or multi-step workflows?

A: Teams should evaluate the full trace, not only the final output, when AI systems call tools or run multi-step workflows. Tool use, retrieval, and intermediate decisions are where many failures emerge, so the evaluation design has to capture those steps, score them consistently, and preserve the evidence for release decisions and incident review.

Q: Why do pre-deployment tests fail to manage production AI risk?

A: Because they only measure behaviour before release. Once a model is connected to live users, tools, and data, the risk shifts to runtime. Prompt injection, unsafe retrieval, and policy bypass happen in execution, so organisations need controls that can intervene while the request is being processed.

Q: When should organisations block AI releases with evaluation gates?

A: Organisations should block AI releases when the system’s behaviour is safety-critical, customer-facing, or tied to governed enterprise workflows. If a bad response, unsafe tool call, or policy violation would create operational or compliance risk, the evaluation result should be treated as a release control, not as a passive report.

Q: What should security teams do after an AI production failure is discovered?

A: Security teams should turn the failure into a reusable regression test, preserve the trace that shows what happened, and make the same case part of the next deployment gate. That converts an isolated incident into a control improvement and keeps the issue from reappearing unnoticed in later releases.

Technical breakdown

CLI-first red teaming vs production-linked evaluation

Promptfoo represents the CLI-first model: tests, assertions, and red-team cases live in YAML beside code and run locally or in CI. That works well when the team wants deterministic, reviewable checks and can treat evaluation like ordinary source control. Braintrust represents the production-linked model: traces, scorers, datasets, human review, and release gates sit on the same data layer, so the same scoring logic can be applied during development and after deployment. The architectural difference is not cosmetic. One model evaluates the prompt before shipping; the other keeps evaluation attached to the runtime evidence that produced the failure.

Practical implication: choose the operating model first, then place evaluation where your release and incident data actually lives.

Trace-level scoring and release gating

Trace-level scoring means evaluating the full sequence of tool calls, retrieval steps, and intermediate actions rather than only the final response. That matters for agentic and workflow-heavy systems because failures often emerge in the path, not just the output. A CI/CD quality gate turns those scores into a release control, blocking merges when thresholds are not met. This is a governance shift from retrospective QA to enforced pre-release policy. Without trace linkage, a team can know a response was bad without knowing which step, retrieval, or tool use made it so.

Practical implication: if your AI system uses tools or retrieval, evaluate traces and enforce thresholds before merge.

Red teaming coverage and security testing depth

The article contrasts broad plugin-based red teaming with custom-code scorers and adversarial datasets. Promptfoo is positioned around OWASP, NIST AI RMF, MITRE ATLAS, and EU AI Act presets, which makes it strong for structured probe coverage and security testing. Braintrust instead leans on custom scorers, human review, and production traces to detect failures in live traffic. That distinction matters because security testing and production observability answer different questions. One asks what an attacker or misuser can provoke; the other asks what the system actually does under real workload conditions.

Practical implication: do not treat red teaming as a substitute for production observability, especially when agent behaviour can change after deployment.

Threat narrative

Attacker objective: The objective is to get unsafe, unreliable, or policy-breaking AI behaviour past development-only evaluation and into production workflows.

1. Entry occurs through the AI application’s evaluation boundary, where prompts, tests, or live inputs are supplied into the system during development or CI.
2. Credential or control abuse appears when tool-using or workflow-linked models act on inputs without production traceability, making failures hard to attribute to a specific step.
3. Impact is the release of behaviour that was not sufficiently tested under real traces, real inputs, or enforced quality gates, allowing flawed AI output or agent actions into production.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI evaluation debt is now a governance problem, not just a testing problem. Once teams ship AI into production, evaluation has to follow the release path, not sit beside it. A local YAML test suite can still be useful, but it cannot govern live traces, live scores, and release enforcement at scale. The practitioner conclusion is straightforward: evaluation that does not touch production is incomplete governance.

Trace-level evidence is the missing control surface for agentic systems. Output-only testing misses tool calls, retrieval decisions, and multi-step workflows, which is exactly where many AI failures happen. When a model can act through tools, the relevant question becomes what it did, not only what it said. The implication for practitioners is to treat traces as control evidence, not as optional observability noise.

Continuous release gating is the right response to AI quality drift. AI systems change through prompt edits, model swaps, retrieval updates, and new tool permissions, so static evaluation quickly goes stale. A gated workflow keeps quality checks attached to the change control process, which is especially important when AI systems interact with identity, access, or other governed enterprise services. The practitioner takeaway is to align evaluation with release policy, not with developer convenience.

Agentic workflows create a new kind of evaluation gap: behaviour can be correct in dev and unsafe in production. That gap is not solved by broader test coverage alone, because the missing element is runtime context and enforcement. When production traces are absent from the evaluation loop, the organisation cannot reliably explain or prevent regressions. The field-level implication is that AI governance now depends on lifecycle control, not just model scoring.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools, according to the Ultimate Guide to NHIs.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Evaluation drift is now a lifecycle risk for AI programmes. Once teams split red teaming, tracing, and release control across separate tools, they lose the ability to prove that a test failure actually blocks a bad deployment. That creates a governance gap between development confidence and production assurance, especially for AI systems that can call tools or touch enterprise data.

Production traces are becoming the real evidence base for AI governance. The control question is no longer whether a prompt set exists, but whether the organisation can reconstruct, score, and reuse what happened in live traffic. Teams that cannot do that will struggle to manage AI quality as a controlled process rather than an ad hoc review activity.

For practitioners

• Define the production evaluation boundary Map which AI workflows are allowed to ship with local-only tests and which require live tracing, scorer reuse, and release gating before merge.
• Tie CI/CD checks to release policy Use automated thresholds that block merges when quality, safety, or security scores fall below the minimum required for deployment.
• Promote production failures into regression tests Turn real trace failures into reusable evaluation cases so the same defect cannot reappear in the next build.
• Separate red teaming from observability Keep adversarial testing focused on probe coverage while maintaining a separate production tracing layer that captures live behaviour and user impact.

Key takeaways

• AI evaluation splits into two models here, one centred on local red teaming and one centred on production traces, and the difference changes how release control is enforced.
• Trace-level scoring matters because many AI failures emerge in tool calls, retrieval steps, and multi-step workflows rather than in the final answer alone.
• Teams need evaluation that feeds release gates and regression tests, or production risk will outrun developer-only testing.

Key terms

• Production Tracing: Production tracing records an agent’s full execution path, including tool calls, retries, and intermediate states. It gives teams evidence about how the system reached a result, which is essential for debugging, evaluation, and governance when final outputs alone do not explain behaviour.
• Approval Gate: An approval gate is the human or policy checkpoint that must be crossed before an AI system can take a state-changing action. For identity governance, the gate is the point where responsibility stays with the organisation, and it should be enforced separately from the system’s ability to observe or recommend.
• Red Teaming: Red teaming is structured adversarial testing used to find how an AI system fails under realistic misuse or attack conditions. In AI security, it is a discovery method, not a proof of safety, because probabilistic behaviour and changing models prevent any lasting guarantee.
• Human Review: Human review is the practice of placing an accountable person between AI output and consequential business action. For culturally sensitive use cases, it is the control that catches misalignment the model cannot reliably detect on its own, especially when tone or social meaning matters.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, human identity, workload identity, and secrets management. It is designed for practitioners who need to connect identity controls to broader security and AI governance programmes.