Compound AI evaluation needs component-level tracing, not final-output checks

At a glance

What this is: This is a guide to evaluating compound AI systems by tracing each step, with the key finding that final-output checks can miss retrieval, ranking, and generation failures.

Why it matters: It matters because AI and platform teams need evidence about where a workflow fails, not just whether the last answer looks acceptable, especially when decisions depend on RAG or multi-step agentic pipelines.

Context

Compound AI systems are AI applications made up of multiple dependent steps, such as retrieval, ranking, tool use, and generation. A single end result can look correct even when one upstream component is underperforming, which makes end-to-end checks too blunt for production governance. For teams running RAG or multi-agent workflows, the question is no longer only whether the answer is right, but whether each step that produced it is behaving as intended.

That creates an evaluation gap that is closely related to AI observability and, where agents or tool chains are involved, to the identity and accountability of each runtime component. Tracing gives teams a way to separate component failure from output quality, which is essential when a pipeline needs to be debugged, benchmarked, or audited across changing data, prompts, and dependencies.

Key questions

Q: How should teams evaluate compound AI systems without relying only on final output checks?

A: Teams should evaluate compound AI systems at the component level, not just by judging the final response. Trace each span, assign metrics to retrieval, ranking, generation, and tool use, then compare those signals against session-level outcomes. That approach reveals where failures begin, which dependencies are weak, and whether a fluent answer is actually grounded in the right evidence.

Q: Why do final-output checks miss the real failure in RAG pipelines?

A: Final-output checks miss the failure because a fluent answer can hide weak retrieval or poor context selection. If the retrieved material is irrelevant, the generator may still produce something plausible, which masks the upstream defect. The risk is misdiagnosis: teams tune the wrong part of the system and leave the actual bottleneck untouched.

Q: What signals show that a compound AI workflow is failing upstream?

A: Low retrieval relevance, poor grounding, and large gaps between span-level and session-level performance are the clearest signals. If the final answer looks acceptable but intermediate metrics are weak, the issue is usually in an upstream component rather than generation. That is the practical value of tracing: it exposes the hidden step where quality broke down.

Q: How do evaluation traces help with accountability in multi-step AI systems?

A: They show which component executed, in what order, and under which runtime context, so teams can connect a failure to the step that produced it. That matters when several services, prompts, or agents contribute to one outcome. Without trace-level visibility, accountability collapses into a single output score that says little about the real cause.

Technical breakdown

Why final-output evaluation misses compound AI failures

In a compound AI system, the final answer is an aggregate of several intermediate operations. Retrieval may return weak evidence, ranking may reorder context poorly, and generation may still produce a fluent response, which hides the failure. That is why output-only scoring often overstates system quality: it measures the last mile, not the path that created it. In RAG and multi-agent workflows, component interactions matter because each step changes the inputs to the next. Tracing and span-level metrics expose where quality degrades, so teams can distinguish a bad model call from a bad upstream decision.

Practical implication: Use component-level evaluation whenever a workflow has more than one material step, so debugging starts at the failing span rather than the final response.

How span-level and session-level metrics work together

Span-level metrics measure a specific operation inside the workflow, such as document retrieval relevance or answer faithfulness at the generation step. Session-level metrics summarize the end-to-end interaction, such as latency, total tokens, or overall consistency against ground truth. The combination matters because session metrics show whether the pipeline is working as a product, while span metrics show where it is failing as a system. This is the difference between operational observability and coarse scorekeeping. When teams instrument both layers, they can compare pipeline versions, isolate regressions, and avoid attributing a downstream symptom to the wrong component.

Practical implication: Attach metrics at both span and session level so you can compare overall behaviour with component health during each experiment or release.

Why RAG pipelines need evaluation at the retrieval boundary

RAG systems depend on a retrieval boundary that is often treated as a black box. If the retriever returns irrelevant documents, the generator may still fabricate a coherent answer, which makes the system appear more reliable than it is. The article’s example shows why semantic similarity on the final response is not enough when the source context itself is weak. The retrieval step needs its own metrics because the pipeline’s truthfulness depends on whether the right material entered the context window in the first place. In practice, this is also where data coverage gaps and vector search weaknesses become visible.

Practical implication: Measure retrieval quality separately from response quality so missing coverage or poor ranking does not get hidden by a polished final answer.

NHI Mgmt Group analysis

Compound AI observability is now a governance control, not just a debugging aid. Once workflows chain retrieval, ranking, generation, and possibly tools or agents, final-output checks cannot establish where trust was earned or lost. The governance issue is traceability across the execution path, because that is where reliability, accountability, and auditability are actually determined. Practitioners should treat tracing as part of AI control design, not a post-hoc analytics feature.

The real failure mode is intermediate blindness. A system can pass end-user checks while still failing on retrieval relevance, grounding, or intermediate logic. That creates a false sense of assurance, especially in compound AI systems where each step can compound the errors of the step before it. The named concept here is intermediate blindness: the inability to see which internal step caused the apparent success or failure. Teams need to eliminate that blindness before they can claim the system is dependable.

Component-level metrics are the only credible way to separate product quality from pipeline quality. A single output score conflates model behaviour, data coverage, retrieval quality, and orchestration performance. That makes root cause analysis slow and often wrong. By splitting evaluation across spans and sessions, teams can align testing with actual system architecture. For practitioners, the implication is clear: reliability programmes for compound AI should be built around measurable components, not just endpoint satisfaction.

Where compound AI intersects with identity and access, the control problem becomes sharper. Multi-step AI systems increasingly depend on distinct runtime components that call data sources and tools under different permissions. If those components are not separately traced, teams cannot tell which identity, credential, or tool invocation produced the action. That matters for AI governance because delegated access without step-level visibility creates an accountability gap. Practitioners should ensure traces preserve the relationship between component behaviour and the privileges used to execute it.

Evaluation strategy is becoming part of AI system architecture. The article shows that teams cannot bolt on evaluation after deployment and expect meaningful assurance. In compound systems, the evaluation model must match the control flow model, including retrieval, generation, and any agentic steps. That is where AI governance matures from generic quality measurement into operational oversight. Practitioners should design evaluation to mirror the workflow, or they will keep measuring the wrong thing.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Intermediate blindness: compound AI systems can appear healthy at the output layer while the actual defect sits in retrieval, ranking, or orchestration. That means evaluation has to move closer to the execution path, because the control that matters most is visibility into the step that created the final answer, not the answer alone.

For practitioners, the immediate signal is architectural: if a pipeline has multiple dependent steps, then every release should be tested against step-specific metrics before end-to-end quality is trusted. This is especially true for RAG and agentic workflows, where a fluent response can conceal poor grounding or tool misuse.

When AI systems depend on delegated runtime components, tracing also becomes part of identity and access governance. Step-level visibility helps teams understand which component used which permission, which is increasingly necessary for auditability in agentic environments.

For practitioners

• Instrument every material step Capture traces for retrieval, ranking, generation, and tool use so each operation can be measured independently instead of only at the final response.
• Separate span metrics from session metrics Use span-level metrics for local quality signals such as retrieval relevance and answer faithfulness, then use session-level metrics for end-to-end latency and consistency.
• Benchmark retrieval before response quality Check whether the retriever is surfacing relevant source material before tuning prompts or generation settings, because weak context makes downstream scores misleading.
• Build evaluation datasets around real workflow failure modes Include cases that expose poor retrieval, weak grounding, and cross-step error propagation so tests reflect how compound AI systems fail in production.
• Add human review for disputed traces Use expert review to confirm whether automated metrics are aligned with domain judgement, especially when a model looks fluent but the retrieved evidence is poor.

Key takeaways

• Compound AI systems need evaluation at the level of retrieval, ranking, generation, and orchestration because a good final answer can hide a broken upstream step.
• Tracing and span-level metrics make it possible to isolate bottlenecks, compare pipeline versions, and stop misattributing downstream symptoms to the wrong component.
• For AI programmes, reliability now depends on evaluation design that mirrors the workflow itself instead of relying on a single output score.

Key terms

• Compound AI system: A compound AI system is a production workflow that uses more than one model, step, or decision point to complete a task. It may route requests, verify outputs, or rewrite results, which means governance must cover the orchestration logic as well as the underlying model calls.
• Component-Level Evaluation: Component-level evaluation isolates parts of an agent workflow, such as retrieval, tool selection, or argument generation, to find precise failure points. It helps teams debug faster and build performance baselines for specific steps. This method is especially valuable when a final answer looks correct but the underlying process is unsafe or unreliable.
• Span: A span is one discrete operation inside a trace, such as a retrieval query, prompt assembly step, or model call. Spans make it possible to pinpoint which part of an AI workflow consumed time, introduced error, or passed bad context to the next stage.
• Retrieval relevance: Retrieval relevance measures whether the documents fetched for a RAG query actually answer the user's question. If relevance is poor, even a well-behaved generator can produce hallucinations because it has the wrong evidence to work from.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and IAM for practitioners building secure identity foundations. It helps security and platform teams connect access control discipline to the wider governance demands of modern digital systems.