Observation debt is the real control gap in AI agent security

At a glance

What this is: This is an analysis of why AI agent prompt injection is hard to control, with observation debt identified as the gap between deployment and enforceable behavioral baselines.

Why it matters: It matters because IAM, PAM, and security teams now have to govern agents that can invoke tools and credentials before traditional approval or review loops can meaningfully constrain them.

By the numbers:

• The UK AISI red-teaming challenge ran 1.8 million prompt injection attacks against 22 frontier-model agents across 44 deployment scenarios.
• Only 44% of companies have implemented any policies to govern AI agents, even though 92% agree governance is critical.

👉 Read ARMO's analysis of prompt injection controls and observation debt

Context

Prompt injection becomes a governance problem the moment an AI agent can follow untrusted instructions, call tools, or act with access that outlives the attack attempt. The core issue is not model intelligence, but whether the agent’s execution path can be constrained before its own behavior has been observed long enough to form a trustworthy baseline.

That creates a direct identity and privilege problem for teams responsible for IAM, PAM, and NHI governance. An agent that can present secrets, reach downstream systems, or trigger state-changing actions behaves like a non-human identity with session-level discretion, so the usual assumptions about static roles, approvals, and periodic review do not hold cleanly.

Key questions

Q: What breaks when AI agents are exposed to prompt injection without runtime baselines?

A: The main failure is that the agent can still complete authorised actions even when the instruction is malicious. If you rely only on text filtering or static policy, the attack moves into valid tool use, where normal access checks still pass. Without a runtime baseline, you cannot reliably tell whether the action sequence is expected or coerced.

Q: Why do AI agents complicate traditional IAM and PAM controls?

A: AI agents complicate IAM and PAM because they can make decisions, chain tools, and act faster than human review cycles can respond. They also blur the line between authentication and authorization, since the same identity may trigger multiple actions after a single approval. That means organizations need policy, telemetry, and revocation designed for autonomous behavior, not just human login events.

Q: How do organizations prove AI agent controls are actually working?

A: Organizations prove control effectiveness by showing which agents accessed which data, what actions they executed, and whether those actions stayed within approved task boundaries. Useful evidence includes logs, policy decisions, anomaly alerts, and review records. Without that chain, governance is mostly declarative.

Q: Who is accountable when an AI agent performs an unauthorized action after injection?

A: Accountability follows the governance model that granted the agent its permissions and execution rights. The owner of the agent workflow, the approver of its tool scope, and the team operating the control plane all share responsibility. Frameworks such as OWASP-NHI and zero trust expect those boundaries to be explicit.

Technical breakdown

What observation debt means for AI agent controls

Observation debt is the period after an agent goes live but before enough of its own runtime behaviour has been recorded to enforce behavioural controls with confidence. During that interval, text filters, delimiters, and approval gates can still reduce opportunistic abuse, but they do not know what is normal for that specific agent. The key distinction is between declared policy and observed policy. Declared policy describes what engineers expected; observed policy captures what the agent actually does across tool calls, destinations, and state changes. For agentic systems, enforcement becomes much safer once the baseline is derived from real execution rather than configuration assumptions.

Practical implication: Start runtime recording before you expect enforcement, and treat the gap as a managed exposure window rather than a one-time rollout task.

Why text filters do not hold against adaptive prompt injection

Prompt injection is an adversarial text problem, but the agentic failure mode is broader than text parsing. Defences such as delimiters, input sanitisation, and detector models can suppress obvious payloads, yet adaptive attackers can rewrite instructions until one variant lands. Once the agent accepts the instruction, the downstream tool calls are still authorised because they come from a legitimate session. That is why prompt injection often bypasses perimeter-style controls while leaving access checks intact. The attack succeeds by moving the abuse into valid execution, not by breaking authentication. The real ceiling is therefore the agent’s residual completable capability, not the number of malicious strings blocked at the input layer.

Practical implication: Use text controls to reduce attack volume, but do not confuse them with a control boundary for privileged agent action.

How per-agent baselines change enforcement from guesswork to control

Per-agent baselines work because they anchor policy in the agent’s own observable behaviour, not in a generic manifest or a broader fleet policy. A support agent, a coding agent, and a remediation agent may all use tools, but their normal destination sets and state-change patterns differ materially. When enforcement is derived from those patterns, the system can tell the difference between a legitimate new workflow and a coerced sequence that stays inside authorised primitives. This is where agent identity and workload identity intersect: the agent is not just a user, but a runtime entity whose permissions, tools, and destinations must be governed as a distinct identity object.

Practical implication: Scope behavioural enforcement per agent and per deployment, then promote from audit to block only after the baseline has stabilised.

Threat narrative

Attacker objective: The attacker wants the agent to complete authorised actions that the user never intended, turning trusted runtime access into a vehicle for data theft, abuse, or disruption.

1. Entry occurs when an attacker supplies prompt injection content through retrieved text, user input, or another untrusted channel that the agent processes as instructions.
2. Escalation happens when the coerced agent follows the payload into legitimate tool use, credential presentation, or state-changing actions that remain authorised at the session level.
3. Impact is realised when the agent finishes a harmful workflow such as unauthorised data access, credential exposure, or destructive actions inside connected systems.

NHI Mgmt Group analysis

Observation debt is the core governance concept this article surfaces. AI agents can reach production before enough of their own behaviour exists to support meaningful behavioural enforcement, which means the control gap is structural rather than accidental. That makes agent rollout a governance problem, not just a model-security problem. Practitioners should treat the time between deployment and baseline settlement as a first-class risk window.

Prompt injection is really an identity-and-privilege problem once agents can act. The article correctly shows that authorised tool use can continue even after the instruction is malicious, because the session itself is valid. That means the security model has shifted from preventing login to constraining what a runtime identity can finish. IAM and PAM teams need to govern agent privileges as continuously exercised capabilities, not static entitlements.

Residual completable capability is the better acceptance concept than attack success rate. Attack success rate says little about whether a coerced agent can actually finish a harmful workflow in your environment. The more useful question is what the agent can still complete after credential residence, tool scope, and approval boundaries are reduced. That aligns with NHI governance because the threat is not just access possession, but what that access still enables under coercion.

Agent governance will increasingly depend on workload identity discipline, not just application policy. Once an agent presents credentials, reaches APIs, and triggers state changes, its identity behaves like a non-human identity with dynamic runtime authority. That brings OWASP-NHI style thinking into agentic AI: credential residence, tool scoping, and lifecycle-bound enforcement become central controls. Practitioners should assume agent identity sprawl will follow agent adoption unless ownership and revocation are explicit.

AI agent security is moving toward audit-first control design because most organisations are not ready to enforce from day one. The operational reality is that enforcement needs evidence, and evidence needs time. That makes staged audit modes, deployment-level attribution, and per-agent promotion paths part of the control plane rather than optional extras. The field should expect agent governance to converge on runtime evidence, not static approval alone.

What this signals

Observation debt will become a standard design constraint for agent programmes. Teams that ship agents faster than they can record behaviour will keep rediscovering the same exposure window. The programme-level response is to make runtime telemetry, deployment-level attribution, and staged enforcement part of the operating model from the start.

AI agent governance is converging with NHI lifecycle control. Once agents can hold secrets, invoke tools, and mutate state, they need the same kind of ownership, revocation, and scope discipline that service accounts require. That is why agent programmes should align with workload identity and NHI governance rather than treating agents as just another application feature.

For practitioners

• Build runtime observation before blocking Instrument agent execution to capture tool calls, destinations, spawned processes, and files accessed so you can derive a per-agent behavioural baseline from live activity rather than from a deployment manifest.
• Reduce residual completable capability Trim what each agent can finish by narrowing tool surfaces, removing unnecessary write paths, and replacing live credentials with dummy or brokered access where the workflow allows it.
• Apply audit mode before enforcement Run the full control logic in audit mode first, review unexplained deviations at the Deployment level, and promote to blocking only after the baseline has stabilised across a deployment cycle.
• Treat agent credentials as NHI lifecycle assets Track which agents hold live secrets, rotate or revoke those credentials on the same lifecycle discipline used for service accounts, and separate declared inventories from runtime-discovered inventories.

Key takeaways

• Prompt injection becomes a control failure when an AI agent can still complete harmful actions inside a valid session.
• The article’s central insight is observation debt, the period before enough runtime behaviour exists to enforce agent policy safely.
• Teams should prioritise per-agent baselines, reduced credential residence, and audit-first enforcement before scaling agent deployments.

Key terms

• Observation Debt: The time between an AI agent entering production and the point at which its own behaviour has been recorded enough to support reliable enforcement. It is a governance exposure window, because behavioural controls depend on real execution evidence, not just declared policy.
• Residual Completable Capability: The set of harmful or sensitive actions an AI agent can still complete after controls such as filtering, approval gates, and credential reduction have been applied. It is a practical measure of what remains possible under coercion, and it is more useful than raw attack success rates.
• Per-Agent Baseline: A runtime profile of normal tool use, destinations, and state changes for one specific agent deployment. It allows security teams to distinguish expected behaviour from coercion or drift, and it becomes the basis for moving from audit mode to enforced controls.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

• The step-by-step observe-to-enforce workflow for AI agents running in Kubernetes and similar runtime environments
• The exact control breakdown for input filtering, credential swapping, Rule of Two, and per-agent audit mode
• The operational thresholds used to judge when an agent's behavioural baseline is stable enough to enforce
• The platform-level telemetry pattern for detecting unusual tool use, destinations, and execution drift

👉 ARMO's full post covers the runtime enforcement path, baseline timing, and agent-specific control trade-offs.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to real-world runtime risk.