Notifications
Clear all
Topic starter
20/08/2026 6:04 pm
TL;DR: Portable agent definitions still need portable observability, because structured traces are what make LLM calls, tool use, and decision paths inspectable across runtimes, according to Arize. The implication is that agent governance now depends on traceability, evaluation, and runtime comparison as core controls rather than optional debugging aids.
NHIMG editorial — based on content published by Arize: Add Observability to Your Open Agent Spec Agents with Arize Phoenix
Questions worth separating out
Q: How should teams govern AI agents that run across multiple runtimes?
A: Teams should govern them with a shared trace schema, consistent evaluation criteria, and clear ownership for tool access.
Q: Why do AI agents need observability if the code is portable?
A: Portability only means the definition can move.
Q: What breaks when teams cannot trace what an AI agent did?
A: Governance breaks first, because teams lose the evidence needed to decide whether the issue was behavior, identity, or control.
Practitioner guidance
- Instrument every agent path with structured tracing Make tracing mandatory for each LLM call, tool invocation, and decision point so you can reconstruct complete execution paths across runtimes and compare them consistently.
- Standardise a runtime-neutral evaluation harness Run the same test set and scoring logic across every supported runtime, then compare output quality, tool usage, and completeness using identical criteria.
- Treat tool access as governed machine privilege Review which tools an agent can call, what data those tools expose, and whether the agent’s permissions are scoped to the minimum needed for the task.
What's in the full article
Arize's full post covers the operational detail this analysis intentionally leaves for the source:
- The exact one-line Phoenix setup used to instrument Agent Spec across runtimes
- The trace list and trace tree views that show how agent, LLM, and tool spans are represented
- The evaluation harness used to compare LangGraph and WayFlow across 10 test inputs
- The code repository and implementation pattern for reproducing the tracing workflow
👉 Read Arize’s post on adding observability to Open Agent Spec agents →
Agent spec observability: what it means for AI governance teams?
Explore further
20/08/2026 6:40 pm
Portable agent observability is now a governance requirement, not just a debugging feature. When an agent can run across multiple runtimes, the risk is not only functional drift but also invisible behavioural drift. Traceability, structured evaluation, and runtime comparison are the only practical way to prove that the same agent still behaves within policy across execution environments. For AI governance teams, this is the point where observability becomes part of control design, not post-incident forensics.
A question worth separating out:
Q: What is the difference between tracing and evaluation in agent governance?
A: Tracing captures what happened during execution, while evaluation judges whether that behaviour met the standard you set. Good governance needs both. Traces provide the evidence, and evaluation turns that evidence into repeatable decisions about release readiness, drift, and control effectiveness.
👉 Read our full editorial: Portable agent observability is becoming a control, not a nice-to-have
