LLM production failures should become regression tests, not ad hoc fixes

At a glance

What this is: This is an analysis of how teams can turn failed LLM production traces into regression tests so recurring hallucinations, retrieval misses, tool errors, and format violations do not reappear.

Why it matters: It matters because IAM, AI security, and governance teams need repeatable controls for AI behaviour, not just incident review, especially where agent workflows touch tools, data, and credentials.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read Braintrust's guide to turning LLM production failures into regression tests

Context

LLM production failures are hard to govern because they often do not trigger traditional application errors. A fluent but wrong answer, a malformed tool call, or a schema violation can look like a completed request unless teams preserve the full trace and treat it as security and quality evidence.

For identity and AI governance teams, the key issue is that agent behaviour can change across prompts, models, retrieval indexes, and tool permissions without leaving a durable control record. That makes production traces a practical source for regression testing, especially where AI systems interact with secrets, data, or privileged workflows.

This pattern is already common in agentic and RAG systems, not an edge case. The operational maturity gap is usually the failure to convert incidents into tests, not the absence of the failure itself.

Key questions

Q: What breaks when production failures are not turned into regression tests?

A: The same defect keeps reappearing because the organisation has evidence of failure but no durable control to prevent recurrence. In agent programs, this usually means prompts, tools, or model changes can reintroduce the same bad behaviour after a release. Without regression coverage, observability becomes diagnosis only, not governance.

Q: Why do production AI traces create security and governance risk?

A: Production traces can include prompts, tool calls, model outputs, and evaluation signals, which means they may expose sensitive context beyond standard application logs. If those traces move into a shared data platform without controls, they can be copied, queried, and reused far more widely than the original operational need justifies.

Q: How do security teams know a regression scorer is working?

A: A scorer is working when it consistently fails the known bad case and passes the known good case under the same conditions. Teams should validate it against human-labelled examples, check for false positives on ordinary outputs, and confirm that it blocks only the intended failure pattern. If the scorer cannot distinguish those cases, it is not a reliable control.

Q: Who should own AI regression tests when agents touch tools and data?

A: Ownership should sit with the team responsible for the model workflow, but governance should include AI security, application engineering, and the control owner for any sensitive data or privileged tool access. When an agent can call tools or retrieve restricted information, the regression suite is part of the control environment, not just the test harness.

Technical breakdown

Why LLM failures evade normal observability

LLM systems can return a syntactically valid response even when the underlying behaviour is wrong. A hallucinated citation, a wrong retrieval chunk, or a malformed JSON object may still be logged as success because the request completed and no runtime exception occurred. That means observability captures service health, but not necessarily answer integrity, tool correctness, or policy compliance. In RAG and agent workflows, the dangerous part is often the hidden intermediate step, not the final message. Production traces therefore need to include inputs, outputs, retrieved context, tool calls, and runtime metadata so failure can be reproduced and tested.

Practical implication: Capture full traces, not just final outputs, or you will miss the evidence needed to build reliable regression tests.

How trace promotion turns incidents into durable tests

A regression dataset works when a failed production span becomes a versioned test case with a label, expected behaviour, and context preserved. The important move is to encode the failure mode, such as hallucination, retrieval miss, tool argument error, instruction-follow failure, or format violation, so the same pattern can be scored later. This shifts incident handling from one-off debugging to repeatable quality control. In practice, the dataset should keep enough metadata to connect the failure to prompt versions, model versions, and deployment changes, because that is what explains recurrence across releases.

Practical implication: Promote diagnosed failures into labelled dataset rows so recurring behaviour can be blocked in CI and monitored in production.

Why scorers matter more than raw examples

A dataset row is only useful if a scorer can check the same failure condition in future runs. Some failures need semantic judgment, such as faithfulness or instruction adherence, while others are deterministic, such as invalid JSON, missing fields, or wrong tool arguments. The scorer defines the control, because it turns a past incident into a machine-checkable rule. That makes it possible to run the same check in CI and live traffic, catching regressions when a prompt changes, a model is swapped, or a retrieval layer shifts. Without a scorer, the dataset is only documentation.

Practical implication: Build scorers for recurring failure modes before release gating, or the regression dataset will not enforce anything.

Threat narrative

Attacker objective: The objective is not classic exfiltration but durable recurrence, where a known failure mode keeps slipping through release controls and reappears in later versions.

1. Entry occurs when a prompt change, model swap, retrieval update, or agent workflow change introduces a behaviour that still appears successful in logs.
2. Credential or policy access is effectively abused when the model produces unsupported claims, wrong tool arguments, or malformed outputs that downstream systems accept as valid.
3. Impact is repeated deployment of the same failure mode into future releases, where the defect keeps resurfacing in production and can propagate into user decisions or automated actions.

NHI Mgmt Group analysis

Production traces are becoming the new control plane for AI quality. LLM failures that only exist in observability dashboards are too easy to forget, reintroduce, or misclassify. The real control is whether a failed span becomes a persistent regression case with a scorer attached. That is a governance shift, not just a testing tactic, because it links production evidence to release gating and auditability.

Trace-to-test conversion closes the gap between AI operations and AI governance. Many teams can diagnose a failure but cannot prove that the same failure will stay fixed across prompt, model, retrieval, or tool changes. Regression datasets make the defect reproducible, while scorers make the control enforceable. For AI security and governance programmes, this is the difference between anecdotal remediation and measurable control coverage.

Regression discipline is now part of agent identity governance. When an AI agent can select tools, access data, or trigger downstream actions, its behaviour becomes part of the identity and privilege boundary. A failed tool call or unsupported retrieval can be a policy failure, not just a model mistake. That makes production-derived regression tests relevant to both agentic AI security and NHI governance, because the system identity must be bounded by behaviour that is continuously revalidated.

Failure-mode labelling is the named concept teams need to operationalise. Labelled failure modes such as hallucination, retrieval miss, tool argument error, and format violation create the search layer that makes regression governance scale. Without that taxonomy, teams end up with piles of traces and no reliable way to cluster or score them. The practitioner conclusion is simple: if the defect cannot be named consistently, it cannot be controlled consistently.

Human review should focus on ambiguous cases, not all cases. High-volume AI systems need automated routing for known failure signatures, with human reviewers reserved for borderline or high-stakes examples. That aligns with broader governance practice in AI RMF style controls: decide what is deterministic, what requires judgment, and what should block release. The result is a control loop that scales without turning review into a bottleneck.

What this signals

Failure-mode labelling is becoming a governance primitive for AI programmes. Teams that can categorise production defects can turn them into measurable controls, while teams that cannot will keep treating the same issues as isolated incidents. The practical signal is that AI quality, security, and compliance are converging around the same trace evidence. For governance teams, the next step is to align regression data with documented AI control ownership and incident response.

Production-derived regression testing also changes how agentic systems should be reviewed under AI governance frameworks such as the NIST AI Risk Management Framework and the OWASP Top 10 for Agentic Applications 2026. The control question is no longer whether an agent can perform a task, but whether the task boundary is testable after every meaningful change. That is a stronger and more auditable position than prompt tuning alone.

Regression testing now needs to account for AI identity boundary drift. When tool use, retrieval, and data access are part of the system design, the control boundary moves with the workflow. That makes continuous trace review, scorer calibration, and release gating part of identity-adjacent governance for AI systems. The teams that prepare for that shift will be better placed to manage agent risk before it becomes a breach or compliance event.

For practitioners

• Preserve full production traces for every diagnosed failure Store the input, output, intermediate tool calls, retrieved context, prompt version, model version, and runtime metadata so the failure can be reproduced later.
• Label the failure mode before you promote the span Use a consistent taxonomy such as hallucination, retrieval miss, tool argument error, instruction-follow failure, or format violation so similar defects can be clustered.
• Convert recurring failures into versioned regression rows Promote representative traces into a dataset only after review, and keep source span IDs in metadata for traceability across releases and incidents.
• Write scorers that test the underlying behaviour Use deterministic checks for schema and tool errors, and semantic scorers for faithfulness or instruction following, then run them in CI and live scoring.

Key takeaways

• LLM failures are often invisible to ordinary observability because the system can appear to succeed while still producing the wrong behaviour.
• Production traces become durable governance evidence only when teams convert them into labelled regression cases with executable scorers.
• For agentic AI programmes, regression testing is a control mechanism, not a QA convenience, because it reduces repeat failures across releases.

Key terms

• Regression Dataset: A regression dataset is a curated set of past failures and representative edge cases used to check whether a proposed fix actually improves a system without reintroducing the original problem. In AI operations, it preserves the evidence that exposed the defect.
• Failure-Mode Label: A failure-mode label is a structured tag that describes what went wrong in a production trace, such as hallucination, retrieval miss, tool argument error, or format violation. It lets teams cluster similar incidents, build targeted scorers, and track whether a control is reducing repeat defects.
• Production Tracing: Production tracing records an agent’s full execution path, including tool calls, retries, and intermediate states. It gives teams evidence about how the system reached a result, which is essential for debugging, evaluation, and governance when final outputs alone do not explain behaviour.
• Scorer: A scorer is the rule or model that assigns a quality value to an agent run against defined criteria. It can judge deterministic checks such as schema validity or more subjective criteria such as task completion, allowing teams to threshold quality before release.

What's in the full article

Braintrust's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step dataset promotion workflows from Logs and BTQL for failed production spans.
• Example scorer patterns for hallucination, tool-call errors, schema violations, and instruction-follow failures.
• Human review and thresholding guidance for deciding which traces should block a release.
• Implementation notes for preserving prompt version, model version, and source-span traceability.

👉 The full Braintrust article covers trace promotion, scorer design, and review workflows in implementation detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps practitioners connect identity control boundaries to the systems, workflows, and privileges they are responsible for.