AI observability is exposing the limits of model oversight

At a glance

What this is: AI observability tracks LLM and agent behaviour in production so teams can detect performance issues, unsafe outputs, prompt injection, and anomalous tool use.

Why it matters: It matters because IAM, PAM, and AI security teams need evidence of who or what accessed which model or tool, under what authority, and whether those actions stayed within policy.

By the numbers:

• Enterprise AI use rose from 55% of organisations in 2023 to 88% in 2025, according to McKinsey’s Global Surveys on the State of AI.
• Generative AI use specifically increased from 33% in 2023 to 79% in 2025, according to McKinsey’s Global Surveys on the State of AI.
• The EU AI Act sets maximum fines of €35 million or 7% of global turnover for failures in high-risk AI governance, according to the article.

👉 Read Lasso Security's analysis of AI observability, agent tracing, and runtime enforcement

Context

AI observability is the practice of continuously measuring how large language models and AI agents behave once they are live. The core governance problem is simple: a model can appear healthy at the infrastructure layer while still producing unsafe, off-policy, or non-compliant outputs, and traditional monitoring rarely sees the full prompt, response, and tool-call chain.

That visibility gap matters for identity security because AI systems increasingly act through service accounts, delegated permissions, and connected tools. When model activity is not tied back to identity, scope, and runtime control, teams cannot tell whether an action was authorised, whether an agent exceeded its intended role, or whether shadow AI is operating outside governance boundaries.

The article’s starting position is typical of organisations moving from pilot to production: they have metrics, but not yet enough context or enforcement to govern behaviour at scale.

Key questions

Q: What breaks when AI observability relies only on pre-aggregated metrics?

A: When observability relies only on pre-aggregated metrics, teams often lose the evidence needed for fast root cause analysis. They can see drift or accuracy decline, but cannot easily trace the issue to a specific input, cohort, or model behaviour. That limitation forces manual investigation outside the platform, slows response time, and increases the chance of repeated errors.

Q: Why do AI agents create more identity risk than traditional LLM applications?

A: AI agents create more identity risk because they can persist state, choose tools, and carry out actions over time. Traditional LLM applications usually produce outputs inside a single request-response cycle, but agents can act across workflows and reuse access. That makes privilege, traceability, and impersonation much harder to govern.

Q: How do teams know if AI observability is actually working?

A: It is working when teams can show which change caused a quality shift, which dataset surfaced the issue, and whether the regression was contained before users were affected. If the team cannot trace behaviour across versions, observability is producing logs, not governance evidence.

Q: What is the difference between AI observability and AI governance?

A: AI observability tells you what the system did. AI governance decides whether it should have been allowed to do it, who approved it, and what happens when it crosses a policy boundary. Observability is a data problem. Governance is an operating model that combines policy, ownership, evidence, and enforcement.

Technical breakdown

How LLM observability captures behaviour, not just uptime

AI observability is broader than infrastructure monitoring. It collects latency, throughput, error rates, token usage, prompts, responses, retrieved context, and model version data so teams can reconstruct what happened at runtime. That distinction matters because an LLM can return a successful HTTP response while still leaking data, following a malicious prompt, or drifting from policy. For AI agents, the same model must also capture tool calls and multi-step execution paths, because risk often emerges in the chain rather than a single exchange.

Practical implication: Instrument full request and response traces, not just service health metrics.

Why prompt injection and goal drift are monitoring problems

Prompt injection works by changing the instructions a model or agent follows, often through retrieved content, user input, or chained tool output. Goal drift appears when an agent starts optimising for an unintended outcome or repeatedly calls tools outside its expected scope. These failures are difficult to catch with static rules because they often look like normal activity until the final result is inspected. Baselines, anomaly detection, and policy-aware alerts are therefore central to observability in production AI systems.

Practical implication: Baselining and anomaly detection should focus on behaviour that violates intended task scope.

How runtime enforcement turns observability into control

Observability alone creates evidence; it does not stop unsafe action. The article’s key architectural point is that monitoring must feed into runtime enforcement such as blocking, redaction, rate limits, and access restrictions on models and tools. That is especially important where AI systems use service accounts or delegated credentials, because an observed policy violation is already a sign that authorisation scope and execution scope are misaligned. In practice, the control loop must connect discovery, detection, and enforcement.

Practical implication: Tie alerts to policy enforcement so unsafe outputs and tool use can be stopped in-session.

Threat narrative

Attacker objective: The attacker aims to manipulate model behaviour or agent actions so the system reveals data, bypasses policy, or takes unauthorised steps on their behalf.

1. Entry occurs when a user prompt, retrieved document, or connected tool introduces malicious instructions into an AI workflow.
2. Credential or tool abuse follows when the model or agent uses delegated access to call APIs, retrieve data, or act beyond intended scope.
3. Impact occurs when the system produces unsafe output, discloses sensitive information, or executes an unauthorised action at production scale.

NHI Mgmt Group analysis

AI observability is becoming the control plane for production AI governance. Once models and agents are in use across business functions, raw uptime metrics are insufficient. Organisations need traceability across prompts, responses, tools, and identities so they can prove what happened and why. The governance challenge is not simply detection, but reconstructability. Practitioners should treat observability as evidence generation for AI risk management, not a dashboard exercise.

Shadow AI creates an identity problem as much as a visibility problem. When employees use personal accounts, browser extensions, or embedded SaaS features, the organisation loses the ability to associate AI activity with sanctioned identity and policy. That weakens both accountability and containment, because there is no clean boundary between approved and unsanctioned use. AI observability therefore needs discovery and inventory capabilities that map AI activity back to governed identities and applications.

Intent-based monitoring should replace metric-only thinking for agentic systems. AI agents do not fail like ordinary applications. They can loop, branch, and complete tasks successfully while still violating scope. A useful concept here is execution-path opacity: when the organisation can see that an agent acted, but not why it took that path or whether the path stayed within policy. Practitioners should design controls around this opacity rather than assume logs alone are enough.

Logging without enforcement produces audit comfort, not risk reduction. The article correctly links compliance logging with runtime policy, because regulators care about both evidence and control. That matters for frameworks such as the NIST AI Risk Management Framework and the EU AI Act, where governance depends on operational monitoring. Teams should align AI observability with formal risk functions, not leave it as an engineering convenience.

AI observability is now adjacent to identity governance even when the primary subject is not IAM. The moment an AI system uses service accounts, delegated access, or connected tools, identity scope becomes part of the control problem. That makes this topic relevant to IAM, PAM, and NHI teams as well as AI security leads. Practitioners should govern AI actions as identity-backed transactions, not as isolated model outputs.

What this signals

AI observability is moving from a monitoring function to a governance requirement because production AI now blends model risk, access control, and operational decision-making. When agentic systems call tools on behalf of users, the control question becomes whether those actions are both explainable and authorised. Teams that already manage IAM, PAM, and NHI inventories should extend those controls to AI runtimes before agent sprawl becomes harder to unwind.

Execution-path opacity: the next AI governance gap is not only whether a model responded, but whether the organisation can explain the tool chain that produced the response. That implies closer alignment between observability platforms, identity systems, and policy engines. For practitioners, the practical move is to treat AI traces as security telemetry and policy evidence, not just debugging data.

For practitioners

• Define AI monitoring objectives before production rollout Set thresholds for latency, drift, unsafe output, and policy violations before the model or agent goes live, so the first production week is not your first security test.
• Log full prompt, response, and tool-call context Capture the complete exchange, including retrieved context, model version, initiating user or service account, and each downstream tool invocation, so investigations can reconstruct what the system actually saw and did.
• Tie observability alerts to runtime policy enforcement Configure blocking, redaction, scope restriction, and rate limiting so a detected violation can be stopped before the next tool call or output is completed.
• Inventory shadow AI and unmanaged integrations Discover AI use across personal accounts, browser extensions, SaaS embeds, and API-connected workflows, then classify which identities and tokens are operating outside approved governance.
• Review AI agent access as identity-backed scope Treat each agent, model, and supporting service account as a governed runtime identity with explicit tool permissions, revocation paths, and review ownership.

Key takeaways

• AI observability is now a governance control, because production models can be healthy while still behaving unsafely or outside policy.
• The hardest risk is execution-path opacity, where an agent’s tool chain is visible only after the fact or not at all.
• Teams need traceability and runtime enforcement together, especially when AI systems operate through service accounts and delegated access.

Key terms

• AI observability: AI observability is the ability to see how AI systems are being used, what information they process, and what actions they trigger. In security programmes, it extends beyond uptime or model quality to runtime visibility, policy enforcement, and audit evidence across human and agent-driven use cases.
• Execution-Path Opacity: Execution-path opacity is the gap between seeing that an AI system produced an outcome and understanding the sequence of steps that led there. It becomes a governance problem when multi-step agents use tools, branches, and delegated access that cannot be reconstructed from basic logs alone.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Agentic Workflow Trace: An agentic workflow trace is a record of the steps an AI agent took during a task, including prompts, retrievals, tool calls, and fallback decisions. It is essential for understanding how a system reached a response, especially when the workflow involves multiple services or policy gates.

What's in the full article

Lasso Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Full monitoring architecture for logging prompts, responses, model versions, and tool calls across LLM deployments
• Runtime enforcement examples for blocking, redacting, and scoping AI actions before they complete
• Operational guidance for discovering shadow AI across browsers, SaaS embeds, and unmanaged accounts
• Practical breakdown of AI agent tracing across multi-step workflows and delegated tool usage

👉 Lasso Security's full post covers the monitoring model, policy controls, and agent workflow tracing in detail

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, IAM, and secrets management. It helps practitioners connect identity controls to the broader security programmes they run across cloud, AI, and operations.