AI agent monitoring exposes a runtime governance gap in production

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Collibra's blog post: Production assurance must expand from model outputs to runtime behaviour, tool use, and policy adherence.

Why it matters: IAM, NHI, and AI governance teams need this framing because agent monitoring is becoming a control plane issue, not just an ML operations concern, and runtime behaviour can outpace traditional review and alerting models.

Context

AI agent monitoring is the practice of watching a deployed system’s runtime behaviour, data access, and output patterns so teams can detect drift before it becomes a production failure. In production, the security question is no longer only whether a model is accurate, but whether the actor stays within its authorised operating bounds.

That shift matters for identity governance because an agent does not merely predict, it acts. Once a system can call tools, retrieve context, and take steps on its own, monitoring has to track action traces, policy triggers, and scope adherence as first-class governance signals.

Collibra’s analysis is typical of the current market discussion: model monitoring is being recast as runtime governance once agent behaviour becomes operationally material.

Key questions

Q: What breaks when AI monitoring only looks at model accuracy?

A: Accuracy alone can hide serious problems such as unfair outcomes, cohort-specific failures, and behaviour changes after deployment. A model can remain statistically strong overall while performing poorly for particular groups or in specific contexts. Effective governance therefore needs explainability, bias analysis, and operational monitoring together, so teams can see whether the model is reliable and acceptable in practice.

Q: Why do AI agents need runtime alerting instead of periodic review?

A: Because agent failures happen inside the execution window, not after it. Periodic review is too slow to catch tool misuse, repeated loops, or scope drift while the agent is still acting. Runtime alerting gives teams a chance to intervene before the behaviour compounds into a broader production or access problem.

Q: What are the signs that agent controls are not working?

A: The clearest warning signs are unexplained tool calls, unexpected privilege use, unreviewed delegated actions, and memory or context changes that alter later decisions. If an agent can move from one system to another without visible policy checks, governance is happening too late. Practitioners should look for behaviour that outpaces their logging and approval boundaries.

Q: How should teams compare model monitoring and AI observability?

A: Model monitoring tells you that a threshold moved, while observability explains why it moved and who should respond. For AI agents, that distinction matters because the useful evidence includes traces, ownership, and data access history. Teams need both, but observability is what turns a warning into a governable incident.

Technical breakdown

Why agent monitoring needs behavioural baselines, not just model metrics

Traditional model monitoring compares live outputs to a baseline built at deployment. That works for prediction systems, but it misses agents because agents can remain numerically stable while changing how they act. Behavioural baselines capture tool-call patterns, step counts, context retrieval, and action sequences, which is where scope drift and policy breaches appear first. In practice, the control boundary shifts from score watching to runtime pattern watching. When an agent starts looping, reaching beyond its grant, or taking a different path to the same output, the model metric can stay flat while governance has already failed.

Practical implication: Practitioners should baseline agent actions and tool use, not rely on accuracy or latency alone.

How drift detection changes when the system can act

Drift is no longer only a data science concept. For models, drift means the relationship between inputs and outputs is moving away from the training baseline. For agents, the drift surface expands to include behaviour, because the system can alter its tool usage, context selection, and action cadence without a visible prediction change. That makes alerting dependent on watching deviations in runtime sequences, not just statistical movement in inputs or outputs. A degraded agent is often detected by what it starts to do differently, not by what it says differently. This is why the control problem becomes operational and identity-aware at the same time.

Practical implication: Teams need drift thresholds for behaviour signals such as tool calls, loop frequency, and scope adherence.

Why observability and monitoring are not the same for AI agents

Monitoring tells you a signal crossed a threshold. Observability tells you why it crossed, who owns it, and what the system was doing at the time. For agents, that distinction matters because the useful evidence includes decision traces, retrieved context, and data access history. Without those artefacts, teams can detect that an agent misbehaved but cannot explain whether the cause was bad context, unsafe tool access, or a broken workflow dependency. Runtime governance therefore depends on pairing signal detection with trace-level context and ownership metadata. Otherwise, the organisation sees a symptom but cannot route a meaningful response.

Practical implication: Practitioners should require trace context and ownership data for every significant agent alert.

Breaches seen in the wild

• Salesloft OAuth token breach: hackers stole OAuth tokens to access Salesforce data via Salesloft.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime monitoring has become a governance layer, not a model quality layer. The article’s real signal is that production assurance now depends on watching how an AI system acts, not only how well it scores. That matters because agent behaviour can stay within acceptable model metrics while still violating policy, scope, or data boundaries. The implication is that identity governance for AI systems must move closer to execution time.

Behavioural drift is the right term for agent risk. Model drift describes statistical movement in inputs and outputs, but agents introduce a second problem: their runtime behaviour can change without the model itself appearing broken. Tool use, context selection, and action cadence can all drift independently of quality metrics. Practitioners should treat that as an identity and governance signal, not just an ML operations problem.

AI monitoring exposes a runtime governance gap that traditional dashboards cannot close. The article shows that the relevant artefacts are decision traces, policy triggers, and scope adherence, not just aggregate performance numbers. That means teams that only monitor scores are blind to the control failures that matter most in production. The practitioner conclusion is straightforward: governance has to inspect actions, not only outputs.

Monitoring for agents collapses the old separation between ML operations and access governance. Once a system can retrieve context, call tools, and execute steps, the security question becomes who or what is authorised to do that work in real time. This creates a bridge between AI operations and identity control that most organisations have not yet formalised. The right response is to treat runtime authorisation as part of the monitoring fabric.

AI observability is now the enclosing discipline for production trust. Monitoring surfaces the breach of expectation, but observability provides the context needed to understand whether the cause was data drift, tool misuse, or a broken decision path. That is why observability is not a reporting layer above monitoring. It is the operational record that makes monitoring governable.

From our research library:

• 7% of security leaders admit they do not know how often their AI systems are making autonomous changes to infrastructure, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Runtime governance now has to follow the action path. A production agent can stay numerically healthy while changing its behaviour, which means teams need controls that watch tool use, loop frequency, and policy triggers rather than relying on model scores alone.

Agent monitoring is shifting identity security toward execution-time evidence. Access review logic assumes a stable actor with a reviewable trail, but agent behaviour is dynamic enough that the most useful signal is the action trace itself. If an organisation cannot see the trace, it cannot govern the outcome.

According to the 2026 Infrastructure Identity Survey, 7% of security leaders admit they do not know how often their AI systems are making autonomous changes to infrastructure. That gap is a monitoring failure as much as a governance one, because unknown runtime change frequency means unknown blast radius.

For practitioners

• Baseline agent behaviour, not just outputs Capture normal tool-call patterns, step counts, context retrieval paths, and action cadence at deployment so later deviation is measurable.
• Alert on policy-trigger and scope-adherence changes Treat policy-trigger rate, scope drift, and repeated tool failures as production signals that can require immediate intervention.
• Route alerts with owner and trace context Attach the decision trace, the system owner, and the risk tier to every alert so responders can judge whether the issue is behavioural, contextual, or access-related.
• Pause agents when behaviour crosses threshold Define an intervention path that can suspend an agent before it completes further actions if it loops, exceeds expected scope, or breaches policy.
• Separate model health from runtime governance Use model metrics for prediction quality and a distinct behavioural view for agent action quality, because one does not reveal the other.

Key takeaways

• AI agent monitoring is evolving from model quality tracking into runtime governance over tool use, scope, and policy adherence.
• The important evidence is behavioural, not just statistical, because an agent can drift without obvious degradation in core model metrics.
• Teams that cannot see traces, owners, and intervention paths will struggle to govern AI systems that act inside production workflows.

Key terms

• Agent Behaviour Monitoring: Agent Behaviour Monitoring is the practice of observing how AI agents actually act in production and turning those observations into measurable controls. It focuses on trajectories, tool use, corrections, and outcomes so teams can detect drift, misuse, and policy violations rather than relying only on pre-launch testing.
• Behavioural Drift: Behavioural drift is the gradual change in what an identity does compared with what it was originally approved to do. For AI agents, drift can come from prompt changes, model updates, expanded integrations, or altered workflows, which makes access review alone an incomplete control.
• Runtime Governance: Runtime governance is the set of controls that verify what a system or agent is actually doing after deployment. It combines monitoring, authorization checks, and access validation so teams can detect drift, misuse, or excessive privilege in motion rather than assuming build-time policy still holds.
• Decision trace: The record of how an access decision was made, including inputs, policy logic, and the final allow or deny outcome. For AI-assisted identity systems, decision traces are necessary for auditability, troubleshooting, and proving that automated access was bounded and explainable.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM, AI governance, or identity security programme, it is worth exploring.