AI observability is exposing the limits of model oversight

At a glance

What this is: AI observability is the practice of tracking production LLM behaviour, costs, prompts, responses, and tool calls, and the article argues that this visibility is now essential to detect drift, unsafe outputs, and agent misuse.

Why it matters: IAM, NHI, and AI security teams need observability because agents and model workflows create runtime access and action paths that traditional logs and point-in-time reviews cannot fully explain or govern.

By the numbers:

• The share of organisations using AI in at least one business function jumped from 55% in 2023 to 88% in 2025, according to McKinsey's Global Surveys on the State of AI cited by Lasso Security.
• Generative AI use specifically went from 33% in 2023 to 79% in 2025, according to McKinsey's Global Surveys on the State of AI cited by Lasso Security.
• Under the EU AI Act, high-risk AI systems must be conformity-assessed, registered, and operational with risk management, data governance, logging, and human oversight by August 2, 2026, with maximum fines reaching €35 million or 7% of global turnover, according to Lasso Security.

Context

AI observability is a governance and security control for systems whose outputs are not deterministic. A model can return a successful response code and still produce hallucinated, unsafe, or policy-breaking content, which means traditional uptime or error monitoring is not enough on its own.

The article's core problem is the gap between adoption and oversight: LLMs and agents are being embedded into business workflows faster than organisations can inventory, trace, and control them. That gap matters directly to identity security because tool calls, API access, and service-account usage become part of the AI control plane.

For IAM and NHI teams, the practical question is not whether to watch AI systems, but where the control boundary sits between human approval, runtime policy, and machine identity execution. The article treats that boundary as already stretched in typical enterprise deployments, not an edge case.

Key questions

Q: How should security teams evaluate AI systems that can act on their own across business tools and data stores?

A: Security teams should treat autonomous AI systems as a distinct access layer, not just another application feature. The core task is to define what the agent may see, what actions it may take, and which systems are off limits. That means least privilege, explicit policy boundaries, and continuous logging across every connected tool before broad deployment.

Q: Why do successful AI requests still create security risk?

A: A request can succeed technically while still producing hallucinations, unsafe content, or out-of-scope actions. In AI systems, success codes do not prove policy compliance. The risk comes from the content, the connected tools, and the downstream action path, so teams need monitoring that inspects behaviour as well as availability.

Q: What are the signs that AI agent observability is failing?

A: The main signs are missing tool metadata, disconnected traces between agent hops, unexplained retries, and an inability to reconcile a successful call with the actual state change it caused. If operators can see that a tool ran but cannot prove what it changed, observability is not yet enough for production control.

Q: How should organisations respond when AI observability reveals shadow AI usage?

A: Treat shadow AI as an unsanctioned access path, not just an inventory issue. First, identify where the model is being used, what data is flowing through it, and whether any service accounts or embedded tools are involved. Then move the workflow under governed policy and runtime enforcement before normal use becomes entrenched.

Technical breakdown

Why model observability needs both telemetry and content inspection

AI observability has two layers. Telemetry covers latency, token use, throughput, uptime, and error rates, which tells you whether the system is healthy. Content inspection adds prompts, responses, retrieval context, tool outputs, and model versions, which tells you what the model actually saw and did. In LLM environments, a request can succeed technically while still leaking sensitive data, producing hallucinations, or following a prompt injection path. Security teams therefore need observability that reconstructs the full decision chain, not just the final status code.

Practical implication: instrument both operational metrics and full prompt-response traces before an AI system goes into production.

How AI agents change the identity and access problem

Agentic systems turn monitoring into an identity problem because they act through tools, APIs, and service accounts over multiple steps. A single request may trigger retrieval, branch logic, multiple tool calls, and follow-up actions, which means the relevant security question is whether each action stayed within intended scope. That is closer to workload identity governance than classic application logging. If the agent can call an API, the access path itself needs traceability, scope control, and revocation logic tied to the task, not just the user.

Practical implication: map agent tool access to the service accounts, tokens, and scopes that actually execute the work.

Why runtime enforcement matters more than alerting alone

Detection without enforcement leaves a gap between noticing risky behaviour and stopping it. The article's model is that observability should feed blocking, redaction, rate limiting, or scope restriction before the next action executes. That is important for shadow AI as well, because unsanctioned tools and browser-based usage can bypass the normal approval process entirely. In practice, observability becomes part of the control plane when it can enforce policy at the gateway, model, or tool layer instead of only writing a log entry after the event.

Practical implication: connect observability alerts to inline controls that can stop unsafe outputs or tool calls in real time.

Threat narrative

Attacker objective: The attacker seeks to manipulate model behaviour or agent actions so the system exfiltrates data, makes unsafe decisions, or performs unauthorized tool-driven work.

1. Entry occurs when a user, prompt, retrieved document, or shadow AI workflow reaches the model with malicious or unintended instructions embedded in the input chain.
2. Credential or tool access is then abused when an agent or model is allowed to invoke APIs, service accounts, or connected systems without sufficiently narrow scope or tracing.
3. Impact follows when the model leaks data, takes an out-of-scope action, or completes a task that appears successful operationally but is unsafe, non-compliant, or attacker-directed.

Breaches seen in the wild

• Salesloft OAuth token breach: hackers stole OAuth tokens to access Salesforce data via Salesloft.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI observability is becoming the operational boundary for AI governance. Once models and agents are embedded in live workflows, the question is no longer whether they are monitored, but whether their actions can be reconstructed and controlled. That shifts the governance burden from static approval to runtime evidence, which is why logging and traceability now matter to both AI security and identity security programmes. Practitioners should treat observability as part of the control plane, not a reporting layer.

Agentic AI turns service-account governance into a first-class AI security issue. When an agent is allowed to call tools, the real identity being governed is the workload identity behind those calls, not just the end user who triggered them. That creates a named gap we can call the runtime identity visibility gap: teams can see that a request succeeded, but not always which identity path, tool chain, or scope made it happen. Practitioners should align monitoring with the identities actually executing actions.

Shadow AI creates a blind zone that traditional inventory processes are too slow to close. Personal accounts, embedded SaaS features, and browser extensions can put model usage outside approved logging and policy enforcement. That is not just a model governance issue, it is an access governance issue because data, prompts, and outputs move through unmanaged channels. Practitioners should assume visibility debt grows faster than policy documentation unless discovery is continuous.

The EU AI Act and similar governance regimes are making observability a compliance control, not a nice-to-have. The article correctly connects logging, human oversight, and auditability to regulatory readiness. That matters because organisations cannot evidence control over AI decisions they cannot trace. Practitioners should build observability to satisfy both security operations and compliance evidence requirements.

Runtime enforcement is where AI security stops being descriptive and becomes preventive. Alerts alone do not change agent behaviour, but blocking outputs, constraining tool scope, and enforcing policy at the gateway can. That is the practical direction of the market: observability systems are converging with control systems. Practitioners should judge tools by whether they can change outcomes, not just explain them.

What this signals

Runtime visibility will become the dividing line between governed AI and merely deployed AI. As adoption rises, the operational problem is no longer whether teams can list every model, but whether they can explain which identities, tools, and prompts each system used. The 7% of security leaders who do not know how often their AI systems make autonomous infrastructure changes illustrate how quickly that visibility debt can outpace governance.

AI observability is turning into a control for identity sprawl, not just model quality. Once agents can call APIs and service accounts, the monitoring problem extends into NHI governance, because every delegated action becomes part of the access story. Practitioners should expect discovery, policy enforcement, and traceability to converge into the same operating model.

Auditability will increasingly depend on runtime evidence rather than policy documents. Organisations that cannot reconstruct prompts, tool calls, and decision paths will struggle to prove compliance or explain an unsafe outcome. The practical signal is simple: if your monitoring cannot answer who acted, what they touched, and why, the governance model is incomplete.

For practitioners

• Instrument full prompt and tool-call traces Capture prompts, responses, retrieved context, model version, and tool outputs so incident review can reconstruct the exact execution path.
• Tie agent actions to workload identities Map each agent, service account, API key, and token to its task scope so tool usage can be traced back to the identity that executed it.
• Enforce policy at runtime, not after the fact Block or redact unsafe outputs, restrict tool access, and apply scope limits before an agent can continue to the next action.
• Continuously inventory shadow AI usage Discover LLMs, browser extensions, personal-account workflows, and embedded SaaS features so unmanaged model activity is not left outside monitoring.
• Red-team after every model or configuration change Re-test prompt injection, jailbreak paths, and policy bypasses whenever a model, prompt, retrieval layer, or integration changes.

Key takeaways

• AI observability closes the gap between model uptime and model behaviour by tracing prompts, outputs, tool calls, and runtime context.
• The real governance challenge is not whether AI systems work, but whether organisations can explain what identities and tools they used.
• Monitoring only matters when it can trigger runtime controls that block unsafe outputs, limit scope, and reduce shadow AI exposure.

Key terms

• AI observability: AI observability is the ability to see how AI systems are being used, what information they process, and what actions they trigger. In security programmes, it extends beyond uptime or model quality to runtime visibility, policy enforcement, and audit evidence across human and agent-driven use cases.
• Agent Workflow: An agent workflow is a machine-driven sequence where an AI system can call tools, access data, and continue execution with limited human involvement. These workflows need governance that sees API activity and tool use, not just browser sessions or user clicks.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Runtime Enforcement: Runtime enforcement is the practice of blocking malicious behaviour while software is running, rather than only detecting it after the fact. It monitors process activity, network actions, and privilege changes so a live attack can be interrupted at the point of execution.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is built for practitioners who need to govern service accounts, workloads, and delegated AI actions with more discipline.