LLM agent design patterns stop control-flow hijack, not residual abuse

At a glance

What this is: ARMO argues that common LLM agent security patterns prevent action hijack but still leave most assistants open to residual manipulation of action content.

Why it matters: IAM and security teams need to treat agent design as only one layer of control, because the real governance problem often sits in what an authorised action carries, not which action gets chosen.

Context

Prompt injection defence in LLM agents is often discussed as a question of whether the model can be tricked into choosing the wrong tool, but that is only part of the problem. In practice, the security gap is frequently in the data carried through an approved action, including recipients, rankings, summaries and file destinations. That is where policy decisions, workflow design and runtime controls meet.

This matters for organisations deploying agentic AI, because a constrained agent can still misuse legitimate permissions if untrusted input shapes the values inside an allowed action. The article’s core point is that control-flow integrity is not the same as end-to-end safety, which is a governance distinction identity and AI teams need to understand.

The pattern is typical of production assistants, not an edge case. Most useful assistants must preserve open output space, and that keeps residual abuse alive even when tool selection is locked down.

Key questions

Q: What breaks when LLM agents can still shape arguments inside approved actions?

A: The control flow may remain intact, but the payload becomes attacker-influenced. That means an agent can still misroute emails, move files incorrectly, alter rankings, or omit important content while technically obeying its allowed action set. The failure is not action selection. It is the absence of governance over the values carried by that action.

Q: Why do LLM agent design patterns reduce but not eliminate prompt-injection risk?

A: They reduce risk because they stop untrusted text from steering the tool choice or sequence. They do not eliminate risk because most assistants still need open output space, and that leaves recipients, arguments, rankings and summaries influenceable inside allowed actions. The remaining exposure is a runtime content problem, not a pure planning problem.

Q: How should security teams judge whether an agent output space is too open?

A: A useful test is whether the task can be reduced to a rigid schema without destroying the product value. If the agent must write free text, choose recipients, order candidates or place files, the output space is open enough for residual abuse. The more open the output, the more compensating controls the runtime needs.

Q: What should organisations do when inherited AI agents already use these patterns?

A: Assume the architecture decision is already made and focus on what remains observable at runtime. Inventory the approved actions, identify the values they carry, and baseline the workload so abnormal arguments or call sequences stand out. Inherited agents need behavioural controls because the design pattern alone does not close the residual.

Technical breakdown

Control-flow integrity in LLM agents

The six design patterns in the article are all aimed at one architectural goal: prevent untrusted input from changing which tool an agent calls or the order in which it calls them. Action-selector, plan-then-execute, map-reduce, dual LLM, code-then-execute and context-minimization all narrow the decision surface before the agent reaches a consequential action. That is genuine control-flow integrity, borrowing a term from systems security. It is a strong property, but it only governs the verb of the action, not the values that flow through it.

Practical implication: Treat these patterns as control-flow controls, not complete prompt-injection prevention.

Residual abuse in approved actions

The article’s most important observation is that a constrained agent can still be steered inside an authorised action. If the agent is allowed to send email, move files, rank candidates or summarise content, untrusted input can still influence recipients, destinations, ordering and omissions. This is not a failure of validation in the narrow sense. The action can pass schema checks and still be harmful because the object inside the action has been shaped by attacker-controlled text. That is why the article repeatedly separates action choice from action content.

Practical implication: Add monitoring and policy for the arguments and outputs of approved actions, not just the tool list.

Rigid schema as the only full closure

The two cases in the article that eliminate the residual do so by removing free text entirely. When the output collapses into a rigid schema, there is nowhere for the injection to steer the response. That works for tightly bounded classifiers or structured extraction tasks, but it reduces usefulness for assistants that need natural-language output. In other words, the safest designs are often the least assistant-like. For most enterprise agents, the design choice is therefore not whether residual risk exists, but whether the workflow can tolerate a narrower output space.

Practical implication: Use rigid schemas where the task permits them, and accept that open-ended assistants need compensating runtime controls.

Threat narrative

Attacker objective: The attacker wants to influence authorised agent actions without needing to hijack the tool selection itself.

1. Entry occurs when untrusted text reaches the agent through email, documents, calendar data, reviews or other retrieved content.
2. Credentialed execution follows when the agent performs an approved tool action with attacker-shaped arguments, rankings or summaries.
3. Impact appears as misplaced files, manipulated recipients, omitted contraindications, boosted rankings or corrupted downstream outputs.

Breaches seen in the wild

• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.
• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Control-flow integrity is necessary, but it is not the same as agent security. The article’s central finding is that six design patterns can prevent injected text from choosing the action, yet they do not govern the values carried by the action itself. That distinction matters because enterprise risk often appears inside an authorised workflow rather than at the point of tool selection. Practitioners should read this as a boundary statement, not a green light.

Residual abuse is the more important governance problem for most deployed assistants. Recipients, rankings, file placements, summaries and omitted sections are all examples of attacker influence that survives a locked-down action path. In identity terms, this is authorised behaviour being pushed into unsafe content by untrusted input. The implication is that agent governance has to cover both the permission to act and the shape of what the action consumes or emits.

Open output space is the real constraint on most production agents. The article shows that only rigid schemas close the residual completely, which is why many assistants remain inherently exposure-prone by design. That does not make them unusable, but it does mean security teams need to stop treating prompt-injection defence as a binary problem. The correct question is which residual each agent design leaves behind and whether the surrounding controls can absorb it.

Agent security needs a runtime discipline, not just an architectural pattern. The article’s strongest operational insight is that design-time controls are inherited by the team running the workload, while the residual is encountered at runtime. That is where behavioural baselines, output scrutiny and access scope become part of the control model. Practitioners should assume that every useful agent leaves some open surface unless the task can be reduced to a rigid schema.

Prompt injection governance should be framed as a trust boundary problem. The article treats direct and indirect injection as one class once untrusted text enters the context window, because the governance issue is what the agent is still allowed to do next. That framing is useful for identity teams because it collapses the debate back to authorised action, delegated trust and output control. The practitioner conclusion is simple: define the boundary around what the agent may carry, not only what it may call.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Residual-abuse governance is becoming the practical issue for agent fleets. Security teams should stop asking only whether a model can be tricked into calling the wrong tool and start asking what an authorised tool call can still be made to carry. That is the control gap most enterprise assistants inherit, and it is why runtime observation must sit alongside design-time patterns.

Open output space is the named concept worth tracking here. When an agent must produce free text, recipients, rankings or placements, there is an open output space that prompt-injection controls cannot fully seal. The consequence for programmes is straightforward: the more natural language the agent must emit, the more the surrounding control plane needs to absorb residual abuse.

The operational signal is whether your AI governance model distinguishes between prevented hijack and permitted misuse. If it does not, incident response will over-focus on tool selection and under-focus on the actual harm channel: the content carried by a legitimate action. That is the gap to close in policy, telemetry and access design.

For practitioners

• Define the residual each agent design leaves open Document whether the open risk is argument shaping, ranking manipulation, output tampering or file placement, then map compensating controls to that specific residual.
• Constrain assistant output to rigid schemas where possible Use structured fields for classification, routing and extraction tasks so the model cannot inject free text into high-risk responses.
• Baseline approved actions at runtime Compare each agent’s actual tool use, arguments and sequence against its own normal behaviour so unusual but authorised calls stand out.
• Separate tool permission from content trust Treat an authorised action carrying attacker-shaped input as a distinct risk case, and apply review to recipients, destinations, summaries and rankings.

Key takeaways

• The article’s core point is that prompt-injection patterns mainly defend action selection, while the values carried by authorised actions can still be manipulated.
• Most real assistants keep an open output space, which means residual abuse persists unless the task can be forced into rigid schemas.
• Security teams should govern AI agents at both design time and runtime, because control-flow integrity alone does not stop authorised misuse.

Key terms

• Control-flow integrity: Control-flow integrity means an agent follows its intended sequence of actions even when it encounters hostile or misleading input. In LLM systems, it reduces the chance that prompt injection can redirect tool choice or execution order, but it does not automatically protect the values carried through those actions.
• Residual abuse: Residual abuse is the security gap that remains after an agent has been constrained enough to resist overt redirection. The agent still performs an authorised action, but the arguments, rankings, recipients, or outputs can be influenced by attacker-controlled content.
• Open Output Space: Open output space means an agent must produce unconstrained or semi-constrained natural language rather than fill a rigid schema. That flexibility preserves usefulness, but it also leaves room for prompt injection to shape what the agent says, sends or emits.
• Rigid Schema: A rigid schema is a tightly structured output format with little or no free-text room. It can eliminate prompt-injection residuals for narrowly defined tasks, because the model has no expressive space left to steer the result.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need a common control language across human, machine, and agent identity programmes.