TL;DR: Portable agent definitions still need portable observability, because structured traces are what make LLM calls, tool use, and decision paths inspectable across runtimes, according to Arize. The implication is that agent governance now depends on traceability, evaluation, and runtime comparison as core controls rather than optional debugging aids.
At a glance
What this is: Arize argues that portable agent observability turns structured traces into a control for inspecting what an agent actually did across runtimes.
Why it matters: It matters because AI governance teams need evidence of tool use, decision paths, and runtime consistency before they can trust an agent in production.
Context
Portable agent specifications let the same agent run across multiple compatible runtimes, but portability creates a governance gap if the execution path becomes opaque. In practice, teams cannot rely on console output or ad hoc logs when an agent is invoking tools, making decisions, and revising its path across different runtimes.
Observability closes that gap by turning each LLM call, tool invocation, and decision point into structured trace data. For AI governance teams, the identity question is not just what the model is, but what runtime, tools, and execution context are being allowed to act on its behalf, and whether those actions remain inspectable and comparable.
The article’s starting point is typical for production AI systems that are becoming harder to reason about as they gain portability and runtime choice.
Key questions
Q: How should AI governance teams use trace data to assess agent behaviour?
A: Treat trace data as the primary evidence for agent governance. Final outputs can look acceptable even when the path to get there is inefficient, inconsistent, or unsafe. Structured traces let teams inspect tool use, retries, decision points, and timing so they can evaluate behaviour, compare runtimes, and detect drift before it becomes an operational problem.
Q: Why do portable agents create governance risk without consistent observability?
A: Portable agents can run in different runtimes while appearing to be the same system, which makes behaviour hard to compare if the evidence format changes. Without consistent observability, teams lose continuity across environments and cannot prove whether a change in runtime altered tool use, workflow quality, or decision logic.
Q: What are the signs that data observability is failing in a production environment?
A: Common signs include recurring data quality issues, slow root cause analysis, inability to trace a defect back to a specific pipeline change, and delays in identifying whether the source is the pipeline or an upstream system. When teams can see an issue only after consumers report it, observability is not providing enough proactive insight to prevent data downtime.
Q: Should organisations compare runtimes by output quality or by trace evidence?
A: They should compare both, but trace evidence is the stronger control because it shows how the output was produced. Output quality alone can hide repeated retries, unnecessary tool calls, or divergent workflows. Trace comparison gives governance teams a repeatable basis for deciding whether a runtime is behaving consistently.
Technical breakdown
How structured traces replace ad hoc logging in agent systems
Structured tracing records each step in an agent execution as discrete spans, rather than flattening the run into a text log. That matters because agent behaviour is not a single prompt-response event. It is a sequence of LLM calls, tool invocations, retries, and branching decisions, each with inputs, outputs, and timing. With OpenTelemetry-style instrumentation, those spans become a machine-readable execution record that can be queried, compared, and evaluated across environments. For governance teams, this is the difference between guessing why an agent acted and proving what happened from evidence.
Practical implication: instrument agent runs so execution data is available as structured trace records, not just human-readable logs.
Why runtime-agnostic instrumentation changes observability governance
Runtime-agnostic instrumentation means the same observability setup follows the agent even when the underlying executor changes. That is important in portable agent architectures because the agent definition is separated from the runtime that executes it. Without common instrumentation, any runtime swap creates a blind spot and undermines comparison. With a shared trace schema, teams can compare behaviour across LangGraph, WayFlow, or other compatible runtimes without changing the governance model. The control objective is consistency of evidence, not consistency of runtime choice.
Practical implication: standardise one trace format across runtimes so platform changes do not break auditability or evaluation.
Why agent evaluation depends on trace data, not outputs alone
Outputs can look acceptable while the path taken to produce them differs materially in quality, cost, or reliability. Traces expose whether the agent used tools, whether it retried appropriately, whether it produced a structured investigation, and whether it followed the intended workflow. That is why evaluation belongs on the captured execution record, not just the final answer. In governance terms, the control is not merely outcome review. It is runtime evidence that lets teams compare behaviour, spot drift, and understand whether the agent is operating within its intended decision boundary.
Practical implication: evaluate agents against trace evidence, because final answers alone do not show how they were produced.
Threat narrative
Attacker objective: The objective in the risk model is not theft but opaque execution, where the agent’s actions cannot be independently traced or evaluated across runtimes.
- Entry occurs when a portable agent is executed in a runtime environment that can invoke tools and make decisions dynamically.
- Credential or authority abuse is expressed as the agent acting through its assigned tool permissions and runtime context, which may differ across environments.
- Impact is loss of governable visibility if teams cannot inspect, compare, or explain the agent’s execution path after the run.
Breaches seen in the wild
- CrewAI Uncrew GitHub token exposure: A CrewAI provisioning error exposed an admin GitHub token for all its private repositories; Noma reported it and CrewAI fixed it in hours.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Portable agent observability is becoming a governance control, not a debugging aid. Once an agent definition can run across multiple runtimes, the control plane moves from code to evidence. The organisation no longer needs just a working agent, it needs a traceable one. That makes structured execution records part of AI governance, because runtime behaviour is what must be reviewed, compared, and explained.
Agent portability creates an accountability gap unless the trace schema is stable. If the same agent behaves differently across runtimes, governance fails the moment the evidence format changes. The important concept here is observability portability: the ability to preserve inspectable execution data even when runtime choice changes. Without that, portability becomes a blind spot rather than an architectural advantage.
Evaluation and observability are now coupled for production agents. The article shows that teams are already moving from informal inspection to programmatic assessment of outputs, tool use, and investigation quality. That is the right direction, but it also means governance teams must treat traces as first-class control evidence, not secondary telemetry. The practitioner conclusion is straightforward: if an agent cannot be traced, it cannot be reliably governed.
AI agent identity is no longer just about who the agent is, but how its actions are evidenced. In portable agent systems, the runtime, tools, and execution context collectively shape what the agent can do. That makes traceability part of the agent identity story, because governance needs to know which actions were taken under which execution context and whether those actions were consistent across environments.
Open standards are doing governance work here, not just integration work. OpenTelemetry and Agent Spec reduce the friction of instrumenting multiple runtimes, but the deeper value is that they create comparable evidence across systems. That aligns with the direction AI governance is already taking: controls need portable proof, not local assurances. Practitioners should design for evidence continuity before they scale portable agents.
What this signals
Observability portability is becoming a design requirement for agent governance. If a team can move an agent between runtimes but cannot keep the same execution evidence, portability creates operational ambiguity instead of control. That means AI governance teams should treat trace continuity as part of their deployment standard, not a post-launch convenience.
The practical question is no longer whether an agent can run somewhere else. It is whether the organisation can still explain, compare, and evaluate its behaviour after the runtime changes. That pushes observability into the same category as policy enforcement and evaluation: a control that must travel with the system.
For teams operating portable agents, the next maturity step is to align runtime choice, tracing, and evaluation under one evidence model. If those three pieces are not connected, the organisation will have more deployment options but less governance assurance.
For practitioners
- Implement structured tracing for every agent run Capture each LLM call, tool invocation, decision point, input, output, and timing so execution is inspectable after the fact.
- Standardise a runtime-agnostic trace schema Use one instrumentation pattern across all compatible runtimes so trace comparisons remain valid when execution backends change.
- Evaluate agents against trace evidence, not final output alone Check whether the agent used tools as intended, followed the expected workflow, and produced evidence-backed results before approving deployment.
- Compare behaviour across runtimes before production rollout Run the same agent definition under each supported runtime and review trace-level differences in latency, completeness, tool use, and decision path.
Key takeaways
- Portable agent observability turns execution traces into governance evidence that can be reviewed across runtimes.
- The main risk is not that agents fail, but that they fail or drift without leaving comparable proof of how they behaved.
- Teams should standardise trace capture and evaluation before scaling portable agents into production.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Portable agents need evidence of tool use and decision paths, which maps to governed identity and privilege use. |
| ASI07 — Insecure Inter-Agent Communication | Runtime-agnostic traces help expose opaque interactions and execution paths in agentic systems. | |
| Recommendation — Instrument agent runs so privilege use and tool actions remain traceable across runtimes. Log agent interactions in a consistent trace format so cross-runtime communication can be reviewed. | ||
| NIST AI RMF | MEASURE — AI Measurement and Evaluation | The article centres on programmatic evaluation of agent behaviour using captured traces. |
| Recommendation — Use trace evidence to measure agent quality, drift, and workflow consistency before production rollout. | ||
| NIST CSF 2.0 | DE.CM-01 — Monitoring for Anomalies and Events | Structured traces provide continuous visibility into agent execution and unexpected behaviour. |
| Recommendation — Treat agent traces as monitoring data and watch for deviations in tool use, timing, and workflow. | ||
| MITRE ATLAS | TA0007 — Discovery | Trace inspection supports discovery of how an AI system is behaving and where control gaps exist. |
| Recommendation — Use trace review to discover unexpected agent behaviour and refine defensive detection logic. | ||
Key terms
- Structured Trace: A structured trace is a machine-readable record of an execution path. In agent systems it captures calls, tool use, decisions, inputs, outputs, and timing so teams can reconstruct behaviour and compare runs across environments.
- Runtime-Agnostic Instrumentation: Runtime-agnostic instrumentation is an observability setup that works the same way regardless of which compatible runtime executes the agent. It preserves comparable execution evidence even when the underlying runtime changes, which is essential for portable agent governance.
- Agent Evaluation Harness: An agent evaluation harness is the infrastructure that runs tests, applies scoring rules, aggregates results, and supports release decisions. It connects datasets, execution, scoring, and monitoring so teams can evaluate agents consistently across development, testing, and production-like environments.
- Observability Portability: Observability portability is the ability to preserve usable telemetry, traces, and evaluation evidence as an agent moves across runtimes. It matters because portability without comparable evidence weakens governance and makes behavioural differences harder to prove.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and workload identity. It gives security and identity practitioners the concepts needed to govern non-human actors as they move through production systems.
Published by the NHIMG editorial team on August 20, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org