Span-Level Evaluation

A method of scoring individual steps inside a larger AI workflow. Each span can be measured for accuracy, compliance, or task quality, allowing teams to see which part of an agent run succeeded or failed and to reuse those results in testing and monitoring.

Expanded Definition

Span-level evaluation is a granular assessment method for AI systems that measures individual spans inside a workflow rather than scoring only the final output. In practice, a span can represent a tool call, retrieval step, prompt-response exchange, policy check, or any other bounded action within an agent run. This makes the term especially useful in agentic AI and LLM operations, where a single failure may be hidden inside an otherwise acceptable end result. Definitions vary across vendors, but the common idea is consistent: break the run into inspectable units, score each unit against a rubric, and preserve the results for test reuse, monitoring, and incident review. For governance context, teams often map the outcome to the NIST Cybersecurity Framework 2.0 to anchor evaluation activity in repeatable oversight.

The concept is different from output-only grading because it exposes where quality, safety, or policy adherence breaks down. It is also different from trace logging alone, because logging records what happened while span-level evaluation interprets whether each step met expectations. The most common misapplication is treating a final “good” answer as proof that every intermediate span was safe or correct, which occurs when teams skip step-level review in complex multi-tool agent flows.

Examples and Use Cases

Implementing span-level evaluation rigorously often introduces annotation overhead and rubric design work, requiring organisations to weigh diagnostic depth against review cost.

• An AI agent searches internal documentation, and each retrieval span is scored for relevance and source quality so weak evidence can be isolated before it affects the final answer.
• A customer support assistant uses multiple tools, and the planning span is evaluated for policy compliance while the response span is checked for tone and factual accuracy.
• A code-generation workflow is measured span by span, with separate checks for package selection, code synthesis, and test execution so regressions are easier to localise.
• A security operations agent is reviewed against control objectives, with spans scored for correct escalation, safe action selection, and avoidance of unauthorised tool use.
• A model evaluation pipeline reuses prior span scores to build regression tests, creating a more stable benchmark than judging only end-to-end success.

For organisations formalising this approach, the NIST CSF perspective is useful because it encourages repeatable assessment and response practices rather than one-off review. Where agentic systems are involved, span-level evaluation is often the difference between knowing that a workflow failed and knowing exactly which action caused the failure.

Why It Matters for Security Teams

Security teams care about span-level evaluation because agent failures are often partial, not total. A workflow may produce a plausible output while still leaking data, violating a policy, or taking an unsafe action in an earlier step. That matters for AI security, but it also matters for identity-bound workflows where an agent uses secrets, APIs, or delegated privileges. In those cases, step-level review helps distinguish harmless variability from genuine control failure. The idea aligns closely with governance patterns in the NIST Cybersecurity Framework 2.0 because the framework expects organisations to understand, manage, and monitor risky behaviour across operational processes.

Span-level evaluation also supports NHI oversight when autonomous software entities act on behalf of users or services. A poorly judged span can turn a routine agent run into an over-privileged action, an unsafe retrieval, or a bad decision that propagates into downstream systems. Organisations typically encounter the operational need for span-level evaluation only after an agent incident, at which point reconstructing the failing step becomes unavoidable to contain the issue and prevent recurrence.

Related resources from NHI Mgmt Group

• How do you know if component-level AI evaluation is actually working?
• Session-Level Evaluation
• Trace-Level Evaluation
• Span-Level Telemetry