Continuous evaluation for AI agents needs live trace classifications

At a glance

What this is: This is a technical explainer on continuous evaluation for AI agents, showing how live trace classifications let teams score production behaviour without creating a separate scorer for every failure mode.

Why it matters: It matters because AI agents can drift outside tested paths after release, and identity, access, and governance teams need controls that detect harmful runtime behaviour before it becomes repeatable risk.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

👉 Read Braintrust's guide to continuous evaluation for AI agents with trace classifications

Context

Continuous evaluation is a production control problem, not a testing convenience. Pre-release evals can confirm that a known dataset behaves as expected, but they cannot anticipate every user request, drift pattern, or failure mode that appears once an AI agent is in live service. That gap matters for AI agent governance because the agent's behaviour changes after deployment, which means risk moves from design-time assurance into runtime observation.

Trace classification helps close that gap by turning live traffic into structured signals that evaluation rules can consume. In identity and access terms, the challenge is similar to governing a non-human identity whose behaviour is not fully predictable at build time. Braintrust's approach is one example of how teams are trying to make runtime scoring manageable without multiplying scorers for every individual issue.

Key questions

Q: How should security teams govern AI systems that learn from production traces?

A: Treat production traces as governed evidence, not disposable logs. Teams should be able to convert failures into repeatable tests, preserve prompt lineage, and tie remediation to release controls. If traces cannot flow into evaluation without manual reconstruction, the governance model is too weak for production AI.

Q: Why do AI agents need runtime evaluation in addition to CI testing?

A: CI testing only covers cases the team already anticipated, while live agents encounter new intents, user inputs, and workflow drift after release. Runtime evaluation catches those emergent failure patterns and turns them into review items or regression tests, which is essential when the production environment reveals risks the test set never represented.

Q: What breaks when continuous evaluation is built before classifications are stable?

A: Scorers become brittle because predicates depend on labels that may change as clusters evolve or topic prompts are refined. The result is noisy scoring, missed matches, and false confidence in the monitoring layer. Teams should stabilise trace classifications first, then write predicates that can survive label drift and taxonomy updates.

Q: How do teams decide whether a scored trace is a governance signal or just noise?

A: Use the score as a triage hint, not a final verdict, and require human review for ambiguous or high-impact failures. A useful governance signal is one that can be tied to a workflow, an owner, and a follow-up action, while noisy signals are usually those without stable labels, clear thresholds, or downstream handling.

Technical breakdown

How continuous evaluation extends beyond pre-deploy and CI gates

Pre-deploy and CI evaluation both operate on curated test cases. They are useful for catching known regressions, but they only cover failures the team already anticipated. Continuous evaluation adds a runtime layer by scoring live traces after production interaction, which makes it possible to detect issues that emerge from real prompts, user behaviour, and data drift. The mechanism is simple: collect traces, classify them, and apply scorer logic only when the trace matches a predicate. That shifts evaluation from a static test gate to an ongoing control over observed behaviour.

Practical implication: teams should treat production trace scoring as a separate control layer, not as a replacement for pre-release testing.

Why trace classifications reduce scorer sprawl

A separate scorer for every failure mode does not scale well in production. Classification-driven scoring reduces that burden by using a small set of labels, such as task, sentiment, or issue type, to group traces before scoring. A single predicate can then cover many instances of the same pattern, for example negative sentiment during checkout or repeated tool misuse in a specific workflow. This is similar to policy-based governance in identity systems: the control is easier to manage when a few stable attributes drive many decisions rather than when every case needs bespoke logic.

Practical implication: define a small set of stable trace facets before expanding scorer coverage across more agent workflows.

Why production scoring needs downstream routing and review

A scored trace is only useful if it triggers action. Continuous evaluation becomes operational when scores feed alerts, human review queues, or regression datasets. That is the difference between measurement and control. It also introduces a governance requirement: the score should indicate likely failure, not serve as an unquestioned verdict. Human review remains necessary where model judgment or classification drift could distort the result. In practice, continuous evaluation works best when it is connected to triage, so a production failure becomes a repeatable quality signal rather than an isolated log entry.

Practical implication: wire every production scorer to a response path before expanding coverage.

NHI Mgmt Group analysis

Continuous evaluation is becoming the production equivalent of identity governance for AI agents. Once an agent reaches live traffic, its risk profile changes from controlled test behaviour to observed runtime behaviour. That makes the trace, not the benchmark, the primary unit of governance. For identity teams, the parallel is clear: non-human identities need monitoring at runtime because build-time policy alone cannot capture how they actually behave in context. Practitioners should treat classification-backed scoring as a governance control, not a quality-of-life feature.

Trace classifications create the named concept that matters here: evaluation rule sprawl. Without a classification layer, every new failure pattern pushes teams toward one-off scorers, which becomes unmaintainable as agent surfaces expand. The article shows a more scalable pattern, where a few labels drive many checks, and that has direct relevance for AI governance debt. The practical conclusion is that teams should standardise trace facets early or accept a growing backlog of brittle, duplicate evaluators.

Runtime scoring changes the boundary between testing and operations. Production failures no longer need to stay as isolated incidents if they can be promoted into regression coverage, which creates a feedback loop between ops, engineering, and governance. That aligns with broader control thinking in NIST AI RMF and NIST CSF, where detection and response are part of disciplined operation, not afterthoughts. Practitioners should use continuous evaluation to connect live agent behaviour to repeatable controls and documented accountability.

The identity angle is not optional when AI agents can act independently across systems. An agent that can access data, invoke tools, and trigger workflows behaves like a governed non-human identity, even if the article focuses on evaluation mechanics. That means the evaluation layer should be read alongside identity lifecycle, privilege scope, and access review, because the same runtime behaviour that creates quality risk can also create security exposure. Teams should align eval rules with identity governance, not keep them in separate silos.

The practical value lies in turning scoring into an operational decision point. Alerts, review queues, and dataset promotion are governance outcomes, not just MLOps conveniences. That is where organisations can measure whether their agent programme is improving or merely generating more telemetry. Practitioners should use continuous evaluation to reduce decision latency between the moment a live trace fails and the moment someone can act on it.

What this signals

Trace classification is becoming a practical control boundary for AI agent governance. As agents move from test environments into live workflows, the question is no longer whether the model passed a benchmark. The question is whether the organisation can classify, score, and route the behaviour that appears only in production. That is where runtime governance starts to resemble identity monitoring, especially when non-human identities can act across multiple systems.

Evaluation rule sprawl is the risk signal hidden inside this pattern. If every failure mode needs a separate scorer, the programme will eventually spend more time maintaining checks than using them. Teams should design around reusable facets, then tie those facets to policies, review queues, and regression promotion. The best programmes will make scoring inexpensive enough to scale without making it loose enough to lose meaning.

For practitioners

• Define stable trace facets first Start with a small set of stable classifications such as task, sentiment, issue type, and any product-specific facet that reflects your highest-risk workflows. Avoid writing scorer logic until those labels are consistent enough to support repeatable predicates.
• Route every scored failure into a response path Connect each online scoring rule to an alert, human review queue, or regression dataset so that production failures produce a concrete follow-up action. A score with no downstream owner creates backlog instead of control.
• Separate runtime scoring from release gating Use continuous evaluation to detect live behaviour, then promote important failures into pre-deploy or CI datasets when they need to block future releases. This keeps runtime monitoring and release assurance aligned without conflating them.
• Review predicates whenever classifications change Recheck scoring rules after topic regeneration, facet prompt updates, or trace taxonomy changes, because a predicate tied to a renamed cluster can silently stop matching. Rewind the affected window before you trust the updated labels.

Key takeaways

• Continuous evaluation closes the gap between curated testing and real production behaviour for AI agents.
• Trace classifications make runtime scoring scalable by reducing the need for one scorer per failure mode.
• AI agent governance improves when every scored trace has an owner, a workflow, and a regression path.

Key terms

• Continuous Evaluation: Continuous evaluation is an ongoing measurement loop that checks whether an AI system still performs correctly as data, language, and requirements change. For regulated workflows, it is essential because static test sets quickly become stale and can hide drift in accuracy or ranking quality.
• Trace Classification: Trace classification is the process of assigning structured labels to an agent trace, such as task, sentiment, or issue type. Those labels let scorers apply rules to groups of traces rather than evaluating every event individually, which reduces scorer maintenance and makes production monitoring more scalable.
• Regression Dataset: A regression dataset is a curated set of past failures and representative edge cases used to check whether a proposed fix actually improves a system without reintroducing the original problem. In AI operations, it preserves the evidence that exposed the defect.
• Scorer Predicate: A scorer predicate is the rule that determines whether a trace matches the failure condition a scorer is designed to detect. It is typically a compact logic check over trace labels or metadata, which makes the scorer reusable across many similar traces instead of requiring custom logic for each case.

What's in the full article

Braintrust's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step scorer implementation examples in TypeScript and Python for production trace evaluation
• Automation settings for sampling rate, trace scope, and scoring-rule configuration in the Braintrust UI
• Practical examples of moving flagged traces into alerts, human review, and regression datasets
• Guidance on tuning predicates when topic labels shift after classifier regeneration

👉 Braintrust's full post covers scorer setup, automation rules, and production triage workflows

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It is designed for practitioners who need to connect identity controls to modern AI and access risk.