Topic starter
21/08/2026 4:53 pm
TL;DR: AI agents that work in notebooks often fail in production because debugging, tracing, and infrastructure management lag behind agent logic, according to Arize. The operational lesson is that agent governance now depends on continuous observability and identity-aware controls, not just model quality or deployment convenience.
Editorial analysis by NHI Mgmt Group, based on content published by Arize: “AWS Bedrock AgentCore Observability with Arize AX: Operationalizing AI Agents At Scale”.
Key questions
Q: How should security teams govern agentic AI as it moves into production?
A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature.
Q: Why do AI agents create access risk even when the model is accurate most of the time?
A: Because the risk is not only incorrect reasoning, it is incorrect action.
Q: What are the signs that AI agent observability is failing?
A: The main signs are missing tool metadata, disconnected traces between agent hops, unexplained retries, and an inability to reconcile a successful call with the actual state change it caused.
Practitioner guidance
- Instrument agent execution end to end Capture prompts, tool calls, model interactions, latency and errors in a single trace so production behaviour can be reconstructed without guessing.
- Constrain agent permissions with least privilege Bind each agent runtime to narrowly scoped IAM roles and authenticated endpoints so access reflects the task, not the platform default.
- Standardise telemetry on OpenTelemetry Use portable trace formats and consistent span naming so observability survives framework changes and cloud migrations.
Bottom line: AI agents that work in notebooks can still fail in production because live traffic, tool latency and runtime complexity expose gaps that local testing hides.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
21/08/2026 5:11 pm
Production observability is now an access-control problem as much as a debugging problem. When an AI agent can call tools, consume models and move across services, visibility into its execution becomes part of governance rather than a separate monitoring layer. The article shows why traceability, evaluation and runtime controls now belong in the same operating model. Practitioners should treat agent observability as evidence of controlled behaviour, not just operational convenience.
A few things that frame the scale:
- 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: Should teams prioritise runtime telemetry or tighter agent permissions first?
A: Prioritise both, but start by defining the permissions boundary because observability cannot reduce the blast radius of an overbroad agent. Then add traceability so you can verify that the boundary is actually being respected in production. Without that pairing, teams can see failures but still fail to contain them.
👉 Read our full editorial: Operational observability for AI agents changes production governance