Observation debt is the real control gap in AI agent security

At a glance

What this is: ARMO says prompt injection defences for AI agents fail early because the controls that actually hold need runtime observation of each agent before they can be enforced.

Why it matters: IAM, NHI, and AI security teams need to treat agent onboarding as a control-gap problem, because enforcement that depends on observed behaviour cannot protect a newly deployed agent on day one.

Context

Prompt injection is a control problem, not just a model-safety problem. The issue is that an AI agent can often be coerced into authorised actions while still appearing to pass ordinary access checks, which makes runtime governance more important than static policy language.

In this article, ARMO focuses on the period after an agent is shipped but before enough of its own behaviour has been recorded to enforce meaningful controls. That gap matters for AI agents, and for any programme treating them as software entities with delegated tools, credentials, and write paths.

The practical question is whether teams are governing the agent they deployed, or the agent they assumed would appear after enough observation has accumulated.

Key questions

Q: What fails when prompt injection controls are deployed before an AI agent has been observed long enough?

A: The control that fails is behavioural enforcement, because it depends on a runtime baseline that does not yet exist. Text filters and allowlists can reduce obvious payloads, but they cannot safely define what a new agent should be blocked from finishing until the programme has recorded enough real execution to know what normal looks like.

Q: Why do AI coding agents create new IAM risk even when prompt injection is addressed?

A: Because prompt injection only affects what the model may be persuaded to do, while the deeper risk is what the agent can already do with inherited credentials. If the agent has cloud roles, cached tokens, or SSH access, it can cause damage without needing a successful prompt attack. Identity scope, not prompt quality, is the main boundary.

Q: What are the signs that an AI agent baseline is not mature enough for blocking controls?

A: Frequent unexplained deviations, changing tool-by-destination patterns, and policies that are still being edited after each deployment are all signs that the agent's normal behaviour is not yet stable. If the baseline keeps moving, blocking controls will either miss coercion or break legitimate work.

Q: Should teams prioritise prompt filtering or per-agent behavioural enforcement first?

A: Prioritise both in sequence, not as substitutes. Prompt filtering and tool allowlists reduce opportunistic attacks immediately, but per-agent behavioural enforcement is the control that actually constrains a coerced agent once enough runtime evidence exists. The right order is exposure reduction first, then enforced baselines when the data supports them.

Technical breakdown

Why text filters decay before agent behaviour is known

Prompt injection defences often start with text-side controls such as delimiters, retrieval filtering, and prompt detectors. These measures inspect what the agent reads, but they do not describe what the agent is allowed to finish once an attacker has shaped the input. The real limitation is timing: the strongest controls in this article depend on knowing an agent's own execution pattern, and that pattern does not exist at deployment. Until enough telemetry is collected, the programme is relying on assumptions about likely behaviour, not evidence from the actual agent.

Practical implication: Treat early text filtering as volume reduction, not as the final control boundary.

Observation debt and per-agent baseline enforcement

Observation debt is the interval between an agent reaching production and the point where enough of its behaviour has been recorded to enforce controls against it. That baseline has to be per agent, because one agent's normal sequence is another agent's anomaly. A static policy generated from a deployment manifest cannot safely capture coerced behaviour if the agent was never observed taking that path in the first place. The result is an enforcement model that must start in audit, learn from real execution, and only later move to blocking mode.

Practical implication: Build the baseline from runtime telemetry, not from expected behaviour written at design time.

Residual completable capability is the better acceptance measure

Attack success rate measures how often an attacker can get a response, which is useful for sizing opportunistic defences but weak as a gate for production readiness. A more durable measure is residual completable capability: given a coerced instance of this agent right now, what can it still finish, what tools can it still reach, and what credentials remain valid. That shifts evaluation from abstract model performance to concrete blast radius. It also aligns with identity governance because the key question becomes which delegated permissions still exist when the agent is manipulated.

Practical implication: Assess the agent's remaining reachable actions before you judge it safe to promote.

Threat narrative

Attacker objective: The attacker wants the agent to complete unintended actions while appearing to operate within valid authorisation boundaries.

1. Entry occurs when the attacker supplies crafted prompt injection content that the agent processes as part of its normal workflow.
2. Credential access and action abuse follow when the coerced agent uses authorised tools, destinations, or credentials that are still within its granted scope.
3. Impact occurs when the agent completes tasks the attacker did not author directly, because the security boundary evaluated only permission and not behaviour.

Breaches seen in the wild

• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Observation debt is the named control gap in AI agent security. Static controls can be deployed immediately, but the enforcement that actually constrains an agent depends on a behavioural baseline that only runtime observation can create. That means the first production risk is not model weakness alone, but the period when teams have already delegated tools and credentials without enough evidence to govern how the agent really behaves. Practitioners should treat this as an issuance-time governance problem, not a later tuning exercise.

AI agents behave like non-human identities with a delayed trust boundary. Once an agent can invoke tools, reach destinations, and present credentials, IAM checks may still return valid even when the sequence itself is unsafe. That is why the identity angle matters: the system is not failing authentication, it is failing to constrain what an authenticated non-human actor is permitted to finish. Security teams should shift from access approval to residual capability control.

Attack success rate is the wrong acceptance lens for agent prompt injection. A benchmark that converges toward certainty after enough retries cannot define production safety, because it measures attacker patience rather than bounded agent behaviour. The more useful concept is residual completable capability, which expresses what an agent can still accomplish after controls are applied. Practitioners should judge agent readiness by how much harmful work remains possible, not by how often an attack eventually lands.

Behavioural enforcement must be sequenced after observation, but blast radius reduction cannot wait. There is no contradiction in starting with text filtering, tool scoping, and credential minimisation while the baseline forms. The governance mistake is assuming those early controls are sufficient when they are really temporary load-shedding. Teams should use them to reduce exposure during the observation window, then promote to per-agent enforcement only when the baseline is stable.

From our research library:

• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.
• Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear value and insufficient risk controls.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Observation debt: AI agent programmes now have a governance interval between deployment and enforceable behaviour, and that interval should be measured as a control exposure rather than treated as normal onboarding friction. Security teams need to plan for that window explicitly, because the agent is already acting before the baseline is ready.

The practical implication is that IAM and NHI teams should think in terms of residual capability, not just approved access. If a coerced agent can still reach live credentials, sensitive tools, or write paths, the programme has not yet reduced blast radius to an acceptable level.

For practitioners

• Establish a runtime observation baseline for each agent Collect execution telemetry for tool calls, destinations, spawned processes, and files opened so enforcement reflects actual behaviour rather than declared intent.
• Scope every agent to residual completable capability Review what each coerced agent can still finish today, including live credentials, reachable destinations, and writable tools, then remove unnecessary reach before moving to blocking mode.
• Run agent policies in audit before enforcement Let the full rule set observe deviations without blocking them, then promote only after the deviation pattern is stable across deployment changes.
• Reduce the attack surface during the observation window Use input filtering, retrieval delimiters, tool allowlists, and approval gates to cut opportunistic payload volume while the per-agent baseline is still forming.
• Promote agents individually against their own baseline Avoid fleet-wide policies that inherit the most permissive agent in scope, and move each agent from audit to enforce on its own recorded envelope.

Key takeaways

• AI agent prompt injection becomes a governance problem when enforcement depends on behaviour that has not yet been observed.
• Runtime baselines matter because static filters and allowlists cannot fully constrain a coerced agent on day one.
• The control objective is to shrink residual capability until behavioural enforcement can safely take over.

Key terms

• Observation Debt: The time between an AI agent entering production and the point at which its own behaviour has been recorded enough to support reliable enforcement. It is a governance exposure window, because behavioural controls depend on real execution evidence, not just declared policy.
• Residual Completable Capability: The set of harmful or sensitive actions an AI agent can still complete after controls such as filtering, approval gates, and credential reduction have been applied. It is a practical measure of what remains possible under coercion, and it is more useful than raw attack success rates.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Per-Agent Behavioral Baseline: A per-agent behavioral baseline is a reference profile of how one AI agent normally acts. It captures typical tool use, request patterns, timing, data access, and decision paths, then flags meaningful deviation. In identity security, it helps detect compromise, prompt injection, privilege abuse, or unintended autonomy in a specific agent runtime.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps practitioners build the control logic that identity and security programmes need as AI agents move into production.