Notifications
Clear all
Topic starter
21/08/2026 4:44 pm
TL;DR: Single-turn scoring can make a chatbot look compliant, polite, and accurate while still missing the real test: whether a full conversation resolves the user’s problem, according to Braintrust. The post shows why multi-turn traces, span-level scoring, and trace-level scoring must work together to catch dropped context, circular exchanges, and unresolved outcomes.
NHIMG editorial — based on content published by Braintrust: How to evaluate multi-turn conversations
Questions worth separating out
Q: How should teams evaluate AI conversations that span multiple turns?
A: They should score both the individual responses and the full conversation.
Q: Why do single-turn evals fail for chatbot governance?
A: Single-turn evals assume the quality of one response is enough to judge the system.
Q: How do you know if a conversational AI system is actually working?
A: Look for evidence that users reach resolution, not just that replies are well written.
Practitioner guidance
- Implement trace-scoped resolution scoring Score the full conversation separately from each turn so you can tell whether the user’s issue was actually resolved, not merely handled politely.
- Keep span-level and trace-level metrics distinct Preserve per-turn quality metrics for wording, tone, and policy compliance, but do not use them as a substitute for outcome-based scoring at the conversation level.
- Instrument conversations as grouped traces Group every related turn under one shared trace ID so downstream analysis can reconstruct the interaction path, compare turns, and identify where context was lost.
What's in the full article
Braintrust's full blog post covers the implementation detail this post intentionally leaves for the source:
- Python code patterns for grouping multi-turn logs under one trace ID and instrumenting turn-level spans.
- The exact scorer setup for brand alignment and conversation quality, including how the LLM-as-a-judge rubric is structured.
- How online scoring rules are configured in the Braintrust UI and how sampling rates affect cost and coverage.
- Topic clustering examples that show how to map low-scoring conversations to recurring issue categories.
👉 Read Braintrust's guide to evaluating multi-turn conversations →
Multi-turn AI evaluations: what single-turn scoring is missing?
Explore further
21/08/2026 5:04 pm
Single-turn evaluation creates a false sense of control: A chatbot can score well on tone and compliance while still failing to resolve the user’s request. That is a governance problem, not just a model-quality problem, because the organisation may optimise the wrong metric and miss the real user outcome. In practice, this is the same control blind spot seen whenever teams audit one step of a workflow but ignore the end-to-end result.
A question worth separating out:
Q: What should teams do when per-turn scores are good but outcomes are poor?
A: Treat that as a measurement failure and redesign the evaluation stack. Keep the per-turn rubric, but add trace-scoped scoring, grouped conversation logging, and topic-level analysis so you can see where the interaction breaks down and why the user still leaves unresolved.
👉 Read our full editorial: Multi-turn evals reveal chatbot failures single-turn scoring misses
