Multi-turn evals reveal chatbot failures single-turn scoring misses

At a glance

What this is: This is a practical guide to evaluating multi-turn AI conversations, and its key finding is that per-turn scores alone cannot prove a chatbot actually resolves user issues.

Why it matters: It matters to IAM practitioners because the same evaluation gap appears whenever AI agents, support bots, or identity workflows must sustain context across multiple steps and decisions.

👉 Read Braintrust's guide to evaluating multi-turn conversations

Context

Multi-turn evaluation matters because a single response score can hide a broken workflow. In conversational AI, the user experience is determined by the whole exchange, not by any one reply, so organisations need a way to measure whether the conversation actually converges on a correct outcome. That same governance problem appears in agentic AI and identity-adjacent workflows, where context, escalation, and resolution must be assessed end to end.

For security and identity teams, the lesson is broader than chatbot quality. Any system that delegates decisions across turns or tasks needs both local and end-to-end visibility, otherwise compliant-looking steps can still produce an unsafe or ineffective result. The article’s example is a support bot, but the evaluation pattern is typical for conversational systems and increasingly relevant for AI agents interacting with governed business processes.

Key questions

Q: How should teams evaluate AI conversations that span multiple turns?

A: They should score both the individual responses and the full conversation. Per-turn scoring catches tone, policy, and clarity issues, while trace-level scoring shows whether the interaction actually resolved the user’s problem. If you only use one layer, you will miss either local quality issues or end-to-end failure.

Q: Why do single-turn evals fail for chatbot governance?

A: Single-turn evals assume the quality of one response is enough to judge the system. In multi-turn interactions, that assumption breaks because the bot can repeat itself, contradict earlier statements, or never close the loop. Governance needs an outcome measure, not just a response measure.

Q: How do you know if a conversational AI system is actually working?

A: Look for evidence that users reach resolution, not just that replies are well written. A working system should complete the task, avoid circular exchanges, and preserve context across turns. If scores are strong on individual replies but weak on conversation completion, the system is not operationally sound.

Q: What should teams do when per-turn scores are good but outcomes are poor?

A: Treat that as a measurement failure and redesign the evaluation stack. Keep the per-turn rubric, but add trace-scoped scoring, grouped conversation logging, and topic-level analysis so you can see where the interaction breaks down and why the user still leaves unresolved.

Technical breakdown

Why single-turn scoring misses conversational failure

Single-turn evals score one output in isolation, so they are good at checking tone, format, or a narrow task like classification. They fail when the system repeats questions, contradicts itself, or appears helpful while never reaching resolution. In a multi-turn interaction, quality is path-dependent: a later turn can correct an earlier one, or a polished reply can still leave the user stuck. That is why conversation quality has to be scored at the trace level, not only at the response level.

Practical implication: measure both turn quality and end-to-end resolution before trusting any conversational system in production.

How trace-scoped scoring changes the evaluation model

Trace-scoped scoring treats the full conversation as the unit of analysis. Each turn remains visible as a span, but the evaluator judges the interaction as a coherent thread, which allows teams to distinguish cosmetic success from actual task completion. This is especially useful in support, onboarding, and agent workflows where the correct answer may emerge only after clarification, escalation, or policy checks. Without trace scope, the system can look healthy while failing its operational purpose.

Practical implication: group related turns under one trace so you can score the conversation as a business outcome, not just as isolated responses.

Why LLM-as-a-judge needs strict scope and instrumentation

LLM-as-a-judge works because a second model can review outputs against explicit criteria, but it only helps if the instrumentation is clean. The article shows that turn spans, trace grouping, and judge spans must be separable so teams can tell generation from evaluation. Otherwise, you cannot explain why a score changed or whether a problem came from the product model, the judge model, or the logging structure. Evaluation quality depends on observability as much as on rubric design.

Practical implication: instrument generation and scoring separately so you can debug the model, the judge, and the conversation structure independently.

NHI Mgmt Group analysis

Single-turn evaluation creates a false sense of control: A chatbot can score well on tone and compliance while still failing to resolve the user’s request. That is a governance problem, not just a model-quality problem, because the organisation may optimise the wrong metric and miss the real user outcome. In practice, this is the same control blind spot seen whenever teams audit one step of a workflow but ignore the end-to-end result.

Conversation-level scoring is a named control concept, not a nice-to-have: The article’s strongest idea is that end-to-end resolution must be measured as its own outcome. That is a useful mental model for agentic AI governance as well, where a sequence of tool calls can look acceptable in isolation and still fail the delegated task. Teams should treat trace-scoped scoring as a distinct control layer, not as a refinement of per-turn grading.

Observable traces are the difference between learning and guessing: The logging pattern described here turns a conversation into inspectable evidence, which is essential for model governance. Without structured traces, teams cannot separate prompt issues, model behaviour, and evaluation error. For practitioners, the lesson is to build evidence trails that support both debugging and accountability.

Multi-turn evaluation aligns better with real operational risk than benchmark-only testing: Benchmarks can reward fluent local answers, but production systems fail at handoffs, clarification loops, and unresolved sessions. That gap matters in customer operations, and it matters even more when AI is embedded in identity or access workflows. The practitioner conclusion is simple: if the workflow spans multiple decisions, the evaluation model must span them too.

Agentic AI governance will increasingly depend on conversation integrity: As AI systems take on more decision-making, the quality of the interaction history becomes a control surface. That makes conversation summaries, trace grouping, and resolution scoring part of the governance stack. Organisations that cannot evidence the whole decision path will struggle to prove safe operation.

What this signals

Conversation integrity is becoming a governance signal for agentic AI: The more a system depends on multi-step interaction, the less useful single-output evaluation becomes. Teams should expect trace-level observability, outcome scoring, and topic clustering to become standard control points for production AI workflows. For adjacent identity programmes, this is a reminder that delegated actions must be assessed as a chain, not as isolated steps.

The operational pattern here is familiar: the control that matters most is often the one that measures whether the workflow finished correctly. That is why AI governance, like identity governance, needs both local inspection and end-to-end assurance. Where the system handles decisions across time, the evidence trail becomes part of the control surface.

For practitioners

• Implement trace-scoped resolution scoring Score the full conversation separately from each turn so you can tell whether the user’s issue was actually resolved, not merely handled politely.
• Keep span-level and trace-level metrics distinct Preserve per-turn quality metrics for wording, tone, and policy compliance, but do not use them as a substitute for outcome-based scoring at the conversation level.
• Instrument conversations as grouped traces Group every related turn under one shared trace ID so downstream analysis can reconstruct the interaction path, compare turns, and identify where context was lost.
• Use topic clustering to prioritise remediation Cluster recurring conversation themes, then inspect low-scoring traces inside the highest-volume buckets first so engineering effort targets the issues creating the most user friction.

Key takeaways

• Single-turn scores can make a chatbot look healthy even when the user never gets a resolution.
• Trace-level scoring adds the missing business-outcome view by measuring the whole conversation, not just each reply.
• Production AI governance needs structured traces, separate judge spans, and topic analysis to turn conversations into actionable evidence.

Key terms

• Trace-scoped scoring: A scoring approach that evaluates an entire conversation or workflow as one unit instead of judging each response independently. It captures whether the interaction reached the intended outcome, which is essential when correctness depends on context, handoffs, or multi-step resolution.
• Span: A span is one discrete operation inside a trace, such as a retrieval query, prompt assembly step, or model call. Spans make it possible to pinpoint which part of an AI workflow consumed time, introduced error, or passed bad context to the next stage.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.

What's in the full article

Braintrust's full blog post covers the implementation detail this post intentionally leaves for the source:

• Python code patterns for grouping multi-turn logs under one trace ID and instrumenting turn-level spans.
• The exact scorer setup for brand alignment and conversation quality, including how the LLM-as-a-judge rubric is structured.
• How online scoring rules are configured in the Braintrust UI and how sampling rates affect cost and coverage.
• Topic clustering examples that show how to map low-scoring conversations to recurring issue categories.

👉 Braintrust's full blog post shows the logging, scoring, and topic-clustering workflow in code.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle control. It is designed for practitioners who need to connect delegated access, workflow integrity, and operational accountability.