Google ADK observability exposes the control gap in multi-agent systems

At a glance

What this is: This is a how-to analysis of tracing, evaluation, and observability for Google ADK, showing how production telemetry turns multi-agent systems from demo-grade orchestration into inspectable agent workflows.

Why it matters: It matters to IAM and security teams because agent decision trails, tool usage, and handoffs increasingly resemble privileged runtime activity that needs governance, not just performance monitoring.

👉 Read Arize's analysis of tracing, evaluation, and observability for Google ADK

Context

Multi-agent systems fail most often at the boundary between orchestration and governance, where a workflow may function in testing but remain opaque once tool use, routing, and handoffs become dynamic. That visibility gap matters because agent behaviour can change per request, per tool, and per state transition, which makes post hoc debugging and access review far less reliable than in deterministic software. In identity terms, agentic systems behave more like runtime identities than static applications, so governance has to follow the action trail, not just the deployment record.

Arize’s example shows the practical side of that problem: routing logic, tool selection, and memory state all become control points that need telemetry if teams want to understand what the system actually did. For IAM, PAM, and NHI programmes, the takeaway is that agent observability is part of access governance when agents can call tools, hand off tasks, and persist state across sessions.

Key questions

Q: How should security teams control autonomous agent actions in production?

A: Security teams should validate every high-impact action at runtime against intent, context, and allowed scope. That means tool invocation controls, not just prompt filters, plus tight identity scoping, logging, and block rules for destructive or data-exfiltrating operations. If the agent can act, the action itself must be governed.

Q: Why do multi-agent systems create governance risk beyond normal application monitoring?

A: They create governance risk because the meaningful security event is often the sequence of actions, not the final answer. A system can appear correct while still taking unsafe routing paths, calling unnecessary tools, or reusing state in ways that widen exposure. Normal monitoring misses that hidden path, so teams need trajectory-level evidence.

Q: How do security teams know if agent observability is actually working?

A: Observability is working only when teams can tie together token activity, tool calls, and latency for a specific agent session. If those signals are disconnected, the logs may show traffic but not governance. The test is whether investigators can reconstruct the path of a meaningful agent action from start to finish.

Q: How do you know if agent evaluation is actually working?

A: It is working when it explains failures well enough to change the system, not just when it outputs a score. Good evaluation should point to the layer that failed, produce a reproducible trace, and feed confirmed failures into regression tests. If the score does not change engineering decisions, it is not yet operational.

Technical breakdown

How ADK routes work across multiple agents

Google ADK uses a root orchestrator to delegate requests to specialist agents based on context, state, and prompt instructions. That model can be deterministic for known phases, but it can also be adaptive when LLM-driven routing decides which sub-agent should act. In practice, the control plane is distributed across prompts, callbacks, tool descriptions, and session state. That makes the system flexible, but also harder to reason about because the same user request can follow different paths depending on context, memory, and available tools.

Practical implication: trace routing decisions and keep policy expectations explicit for every handoff path.

Why observability must capture tool calls and handoffs

In multi-agent systems, the risk is not only what the model says. It is also which tools it invokes, what data it passes, and how one agent delegates to another. OpenTelemetry-style tracing turns those interactions into structured evidence that teams can inspect after the fact. Without that layer, you cannot reliably separate model error from tool misuse, state contamination, or a bad routing decision. For security teams, that distinction matters because each handoff can expand the blast radius of a compromised prompt, misconfigured tool, or excessive permission set.

Practical implication: instrument every tool invocation and inter-agent transfer before promoting agents to production.

Why evaluation is different for agentic workflows

Standard LLM evaluation checks output quality, but agentic systems need trajectory-based evaluation. That means measuring whether the agent chose the right sub-agent, selected the right tool, and followed a logically valid sequence of steps. In the article’s example, online evaluations flag regressions and feed curated datasets for review, which is closer to a continuous control loop than a one-time test. This is especially relevant when agents can act autonomously enough to reach out to tools, but not predictably enough to trust a single prompt test as a proxy for production behaviour.

Practical implication: evaluate trajectories and handoff quality, not just final responses.

NHI Mgmt Group analysis

Agent observability is becoming an identity control, not just an engineering convenience. Once a multi-agent system can route tasks, call tools, and retain state, it starts behaving like a runtime identity that needs traceability. The governance question is no longer only whether the system answered correctly, but whether its actions were authorised, inspectable, and attributable. That is where observability intersects with IAM and NHI governance, because every tool call is effectively a privilege use event. Practitioners should treat agent traces as governance evidence, not optional telemetry.

Trajectory evaluation is the named concept that matters here. A trajectory is the full path an agent takes through decision, handoff, and tool execution, and that path is what reveals whether the system behaved safely. Output-only testing misses most of the governance risk because two responses can look similar while one involved unsafe routing, unnecessary tool use, or poor escalation logic. For security architects, trajectory evaluation is the closest analogue to control validation for non-deterministic systems, and it should sit alongside access review and logging requirements.

Tool descriptions are security-sensitive because they shape agent privilege selection. In agentic systems, the way a tool is described can influence whether an agent uses it, ignores it, or misuses it. That makes tool metadata part of the control surface, especially when external systems, search functions, or payment actions are involved. The relevant governance lesson is that least privilege for agents is not only about permissions, but also about reducing ambiguous affordances that drive overuse. Practitioners should align tool design with explicit policy and audit requirements.

Production observability closes the gap between prototype confidence and operational trust. The article reflects a broader pattern in agentic AI: teams often prove that something works before they prove that they can govern it. That gap is dangerous when agents can take actions across multiple systems without deterministic paths. Frameworks such as the OWASP Agentic AI Top 10 and NIST AI Risk Management Framework are relevant because they push teams toward accountable design, measurable behaviour, and continuous monitoring. Practitioners should use observability to convert experimental agent behaviour into governable system evidence.

Multi-agent governance will increasingly depend on cross-functional control ownership. The technical boundary between model development, orchestration, identity, and monitoring is already blurred, and that is where control failures hide. Security teams need to own the visibility and privilege model, while platform teams own the orchestration mechanics and evaluation loop. Without that split, agent systems will continue to accumulate hidden risk in prompts, callbacks, and delegated tool chains. Practitioners should define who signs off on agent behaviour before deployment, not after the first incident.

What this signals

Agentic systems will push identity programmes toward action-based governance, where traces, tool calls, and handoffs become evidence of control rather than secondary telemetry. That shift aligns with the logic in the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework, both of which reward measurable behaviour over assumed trust.

Trajectory governance: this is the control idea practitioners should watch. The issue is not whether an agent can complete a task, but whether the full decision path can be explained, bounded, and reviewed before it affects production systems. Teams that cannot reconstruct trajectories will struggle to prove least privilege, detect misuse, or support incident response.

For practitioners

• Instrument every agent handoff Capture routing decisions, sub-agent transfers, and tool invocations with structured traces so that production behaviour can be reconstructed after the fact.
• Define evaluation sets for agent trajectories Build regression datasets that score handoff quality, tool selection, and sequence validity rather than relying on response-only testing.
• Review tool metadata as a control surface Treat tool descriptions, permissions, and default instructions as part of the security design, because they influence what an agent chooses to do.
• Separate platform and governance ownership Assign one team to operate orchestration and another to define monitoring thresholds, escalation rules, and approval criteria for production agents.

Key takeaways

• Multi-agent observability is now part of governance because agent traces reveal how decisions, tools, and handoffs actually occurred.
• Trajectory evaluation matters more than output-only testing when systems can route, delegate, and use tools dynamically.
• Security teams should treat agent telemetry, tool metadata, and handoff policy as control evidence before production rollout.

Key terms

• Agent Trajectory: The sequence of intermediate steps an AI agent takes while trying to complete a task, including reasoning, tool use, and decision timing. Evaluating trajectory matters because a system can appear correct at the end while taking unsafe, inefficient, or unauthorized actions along the way.
• Tool Selection Accuracy: Tool selection accuracy measures whether an agent chooses the right tool for a given task, not merely whether the tool responds successfully. It is a governance signal because repeated wrong-tool choices can create cost, security, and compliance problems even when the end output appears acceptable.
• Agent Handoff: Agent handoff is the point at which a system passes an unresolved case from structured automation to an AI agent. The handoff should occur only after the workflow has collected context, because the agent is then extending evidence rather than inventing a case from scratch.
• Observability: Observability is the ability to understand the internal state of a system from the data it produces. In security and operations, that means combining logs, metrics, and traces so teams can explain why something happened, not just confirm that something changed.

What's in the full article

Arize's full post covers the operational detail this post intentionally leaves for the source:

• Step-by-step Google ADK agent orchestration examples for a multi-agent travel concierge.
• OpenTelemetry registration and instrumentation code for capturing agent traces in Arize AX.
• Prompt Playground workflow examples for comparing prompts, models, and evaluation strategies.
• Regression dataset handling and human review loops for production agent tuning.

👉 Arize's full post covers the travel concierge architecture, trace inspection, and evaluation workflow examples.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners apply governance discipline to systems that act at runtime.