Continuous evaluation for AI agents - are your controls keeping up?

TL;DR: Curated pre-deploy and CI evals miss the edge cases that appear only in production, so Braintrust argues that continuous evaluation should score live traces using trace classifications to keep pace with real user behaviour. The operational challenge is less about writing more scorers than about turning production failures into repeatable checks without creating evaluation sprawl.

NHIMG editorial — based on content published by Braintrust: How to build continuous evaluation for AI agents with trace classifications (2026)

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

Questions worth separating out

Q: How should security teams govern AI systems that learn from production traces?

A: Treat production traces as governed evidence, not disposable logs.

Q: Why do AI agents need runtime evaluation in addition to CI testing?

A: CI testing only covers cases the team already anticipated, while live agents encounter new intents, user inputs, and workflow drift after release.

Q: What breaks when continuous evaluation is built before classifications are stable?

A: Scorers become brittle because predicates depend on labels that may change as clusters evolve or topic prompts are refined.

Practitioner guidance

• Define stable trace facets first Start with a small set of stable classifications such as task, sentiment, issue type, and any product-specific facet that reflects your highest-risk workflows.
• Route every scored failure into a response path Connect each online scoring rule to an alert, human review queue, or regression dataset so that production failures produce a concrete follow-up action.
• Separate runtime scoring from release gating Use continuous evaluation to detect live behaviour, then promote important failures into pre-deploy or CI datasets when they need to block future releases.

What's in the full article

Braintrust's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step scorer implementation examples in TypeScript and Python for production trace evaluation
• Automation settings for sampling rate, trace scope, and scoring-rule configuration in the Braintrust UI
• Practical examples of moving flagged traces into alerts, human review, and regression datasets
• Guidance on tuning predicates when topic labels shift after classifier regeneration

👉 Read Braintrust's guide to continuous evaluation for AI agents with trace classifications →

Continuous evaluation for AI agents - are your controls keeping up?

Explore further

View Full Forum →  |  NHI Foundation Course →