Topic starter
20/08/2026 5:57 pm
TL;DR: Curated pre-deploy and CI evals miss the edge cases that appear only in production, so Braintrust argues that continuous evaluation should score live traces using trace classifications to keep pace with real user behaviour. The operational challenge is less about writing more scorers than about turning production failures into repeatable checks without creating evaluation sprawl.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “How to build continuous evaluation for AI agents with trace classifications (2026)”.
Key questions
Q: What breaks when AI agent evaluation relies only on static test cases?
A: Static test cases miss behaviour that appears under load, ambiguity, or changing context.
Q: Why do live trace classifications matter for AI governance?
A: They give teams a structured way to decide which production interactions should be scored, reviewed, or promoted into regression tests.
Q: How do security teams keep online scoring from becoming evaluation sprawl?
A: Start with a small, durable set of trace facets and let scorers read those labels instead of building a new scorer for every failure mode.
Practitioner guidance
- Define stable trace facets first Establish a small set of trace classifications such as task, sentiment, and issue type before writing production scorers.
- Separate trace scope from span scope Use trace-level scoring for multi-turn conversations and span-level scoring for single operations or tool calls.
- Route every flag to a downstream action Connect each online scoring rule to an alert, a review queue, or a regression dataset so a failed trace cannot sit idle.
Bottom line: Production AI agents need live evaluation because curated test sets cannot anticipate every user path, edge case, or drift pattern.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:37 pm
Live evaluation is the missing control plane for AI agent quality. Pre-release test sets can validate known behaviour, but they cannot absorb the volume of real user variability that appears after deployment. Classification-driven scoring turns production traces into governable evidence, which is the only practical way to keep evaluation aligned with actual agent use. For practitioners, the lesson is that runtime observability is now part of AI control design, not an optional add-on.
A few things that frame the scale:
- 62% of organisations are at least experimenting with AI agents, but only 23% have scaled one into a single business function, according to McKinsey's 2025 State of AI survey.
A question worth separating out:
Q: What should teams do after a production trace is flagged by a scorer?
A: Assign each flagged trace a clear next step, such as alerting, human review, or promotion into a regression dataset. A score without a downstream action only creates backlog. The value of continuous evaluation comes from making production failures part of an operational feedback loop.
👉 Read our full editorial: Continuous evaluation for AI agents needs live trace classifications