A span waterfall is a visual timeline that shows each step in an AI agent session in the order it occurred. It helps investigators inspect durations, tool calls, intermediate responses, and dependencies between actions. This makes complex agent behaviour easier to reconstruct than flat logs or summary metrics.
Expanded Definition
A span waterfall is a session-level timeline view for AI agents. It arranges actions in execution order so investigators can see when a prompt was received, which tools were called, how long each step took, and where dependencies or pauses appeared. In practice, it is closer to an execution trace than a dashboard summary.
The term is most useful when an agent performs multi-step work across tools, models, or services. It helps distinguish the agent’s internal sequence from the user-visible result, which matters when a failure happens after several apparently successful steps. A common boundary mistake is to treat the waterfall as a verdict on correctness; it is not. It is a reconstruction aid that makes behaviour legible, especially when logs are fragmented or too coarse to show causality.
Guidance versus consensus: the exact layout and naming are not standardised across vendors, but the underlying need to preserve ordered execution detail is broadly consistent across agent observability tools.
Examples and Use Cases
Span waterfalls typically appear in investigation and debugging workflows where the timing and dependency chain matter more than the final answer alone. They are especially useful when multiple tool invocations or model calls are interleaved.
- Tracing an agent that retrieves data, reasons over it, then calls a second tool before responding.
- Comparing slow spans to identify whether latency comes from model time, tool latency, or repeated retries.
- Inspecting a failed workflow to see whether a tool call returned an error that the agent handled incorrectly.
- Reviewing chained actions in a support or operations agent where one step depends on the output of the previous step.
- Reconstructing an incident when summary telemetry shows success, but the ordered spans reveal an unsafe intermediate action.
For identity-sensitive workflows, a span waterfall can also show whether authentication, token use, or privilege-dependent calls happened in the expected order. That matters because timing and dependency order often explain why an agent was able to act at all.
Security Implications
When span waterfalls are missing, truncated, or too abstract, teams lose visibility into how an agent reached a harmful state. The result can be incomplete incident reconstruction, weak root-cause analysis, and false confidence from tidy summaries that hide repeated tool use, failed guardrails, or unintended branching.
Security problems often emerge at the seams between spans: a benign prompt leads to a tool call, the tool returns sensitive data, and the agent later propagates that data into an output or a downstream action. A waterfall makes those transitions visible. It can also expose misordered controls, such as policy checks that occur after side effects instead of before them.
Another practical failure mode is over-reliance on aggregate metrics. A low average latency or a successful final response can conceal a span that accessed the wrong resource or retried in a way that amplified exposure. Practitioners should read the timeline as evidence of process, not proof of safety.
Domain and Governance Relevance
Span waterfalls matter most in agentic AI governance because autonomous behaviour is only understandable when the full execution chain is visible. They support accountability by showing who or what initiated a step, which tool boundary was crossed, and how long the agent remained in each state. That is especially important when an agent acts on behalf of a user or service account.
For Non-Human Identity operations, the waterfall can help confirm whether an agent used the right credential at the right time, whether a token was reused beyond its intended scope, or whether a privileged action lacked a clear initiating span. That makes the trace useful for ownership, auditability, and boundary verification across identity, tool, and model layers.
In broader governance terms, the value is not just observability. It is evidentiary: a good span waterfall gives teams a repeatable way to explain agent behaviour to security, audit, and engineering stakeholders without relying on memory or summary dashboards.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, OWASP Non-Human Identity Top 10 and MITRE ATLAS address the attack and risk surface, while NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A1 — Agentic Access Control | Span waterfalls expose ordered agent actions and tool boundaries. |
| Recommendation — Review span timelines to verify each tool call stayed within intended agent authority. | ||
| OWASP Non-Human Identity Top 10 | NHI-01 — Inventory and Ownership | Agent spans can reveal which non-human actor used which credential and when. |
| Recommendation — Trace spans to confirm ownership and lifecycle control of agent credentials. | ||
| NIST CSF 2.0 | DE.CM-8 — Monitoring for Anomalies | Span waterfalls improve detection of unusual sequencing, retries, and side effects. |
| Recommendation — Use span timelines to detect anomalous agent execution patterns and control failures. | ||
| CIS Controls v8 | 8.6 — Audit Log Management | Waterfalls support investigation by preserving ordered execution evidence. |
| Recommendation — Retain ordered span records so investigators can reconstruct agent activity. | ||
| MITRE ATLAS | ATLAS-ATTACK-0045 — Tool Misuse | Ordered spans help spot adversarial use of agent tools and chained actions. |
| Recommendation — Map suspicious span sequences to tool-misuse patterns and hunt for abuse. | ||
Related resources from NHI Mgmt Group
- How should security teams govern SaaS access when identities span many apps?
- How should security teams govern AI applications that span notebooks, pipelines, and runtime services?
- Who should own cryptographic trust when machine identities span multiple teams?
- How should security teams replace shared secrets for workloads that span multiple clouds?
