LLM agent design patterns stop control-flow hijack, not residual abuse

At a glance

What this is: This article argues that six common LLM agent design patterns stop injected text from changing an agent’s action, but leave a usable residual in the values that flow through approved actions.

Why it matters: For IAM, NHI and AI governance teams, the key issue is that control-flow integrity is not the same as workload trust, so runtime authorisation and output handling still need governing.

👉 Read ARMO's analysis of LLM agent design patterns and prompt injection residuals

Context

Prompt injection in LLM agents is a governance problem, not just a filtering problem. If untrusted text can shape what an agent does with approved access, then the real risk sits in the action payload, the output space, and the credentials the agent is already allowed to use. In identity terms, that creates a gap between permitted delegation and safe delegation, especially where AI agents are operating with non-human identities, API access, or workspace permissions.

The article’s core claim is that design patterns can preserve control flow without eliminating exploitability. That matters because many enterprise assistants are built to read untrusted content, act on behalf of users, and then carry those results into email, search, file placement, or downstream agent workflows. The security question is therefore not whether the model saw the injection, but whether the surrounding architecture still lets the injected content influence a permitted action.

Key questions

Q: What breaks when LLM agents can still shape arguments inside approved actions?

A: The control flow may remain intact, but the payload becomes attacker-influenced. That means an agent can still misroute emails, move files incorrectly, alter rankings, or omit important content while technically obeying its allowed action set. The failure is not action selection. It is the absence of governance over the values carried by that action.

Q: Why do LLM agents complicate least-privilege IAM models?

A: Because least privilege only works if the allowed action is also safe in context. An LLM agent can act within its granted permissions and still produce harmful outcomes when untrusted input changes recipients, summaries, or destinations. Security teams therefore need both narrow permissions and runtime checks on what those permissions are used for.

Q: How do security teams know whether an LLM agent pattern is actually reducing risk?

A: Look for whether the design reduces the agent’s usable output space, not just whether it blocks obvious tool misuse. If the agent can still write free text, rank items, or choose among sensitive values, the residual is still present. Measure success by comparing approved actions, argument patterns, and downstream effects against the agent’s own historical baseline.

Q: What should teams do when a deployed agent still needs open-ended output?

A: Treat the open-output requirement as an accepted risk and surround it with layered controls. Use least privilege, schema validation where possible, outbound filtering, and behavioural monitoring. If the output is too unconstrained to govern, the safer decision may be to redesign the task rather than assume prompt-level defences will hold.

Technical breakdown

How control-flow integrity works in LLM agents

The six patterns discussed in the article are architectural constraints on what an agent can decide to do next. Action-selection, plan-then-execute, dual-model separation, map-reduce, code-then-execute, and context minimisation all aim to prevent untrusted input from steering tool choice or execution order. In security terms, they create control-flow integrity, meaning the injected text cannot arbitrarily redirect the agent into a forbidden action. That is materially better than relying on detection alone, because the boundary is enforced by design rather than by inference after the fact.

Practical implication: Treat these patterns as control-flow controls, not full abuse prevention.

Why the residual survives in approved action arguments

The article’s central insight is that most patterns do not govern the values carried by an approved action. An agent may still send the wrong recipient, move a file to the wrong place, rank a resume higher, omit a contraindication, or summarise untrusted content in a distorted way. Those are not failures to choose the action. They are failures to govern the object of the action. That distinction matters because validation can confirm that a field is well-formed without confirming that it is safe in context. The residual is therefore an argument-level problem, not only a prompt-level one.

Practical implication: Add argument validation, behavioural baselines, and egress controls around every approved agent action.

Why rigid schemas close the injection path completely

The paper’s strongest results are the cases where the output space collapses into a rigid schema. Once the agent can only populate structured fields, there is no free text left for an attacker to influence, so the prompt injection surface disappears inside that specific task. The trade-off is obvious: the stricter the schema, the less useful the assistant becomes for open-ended work. That is why these designs fit classifiers and tightly bounded workflows better than copilots, research assistants, or coding agents. Security improves as output freedom shrinks.

Practical implication: Use rigid schemas for narrow tasks, but do not expect them to fit general-purpose assistants.

Threat narrative

Attacker objective: The attacker wants the agent to abuse its own authorised actions so the system produces harmful outcomes without ever leaving its approved control flow.

1. Entry occurs when the attacker places malicious instructions inside content the agent is already authorised to read, such as documents, calendar data, emails, or reviews.
2. Escalation happens when the agent keeps its approved tool access but allows the injected content to alter arguments, rankings, recipients, or summaries inside a permitted action.
3. Impact follows when the agent performs a legitimate action with attacker-shaped values, causing misrouting, omission, data exposure, or downstream trust contamination.

NHI Mgmt Group analysis

Control-flow integrity is necessary, but it is not an identity control. The article shows that the strongest design patterns can stop injected text from choosing a new action, yet still leave the agent free to misuse its permitted one. That means the governance problem shifts from prompt defence to entitlement governance, runtime supervision, and auditability of the action payload. For NHI and agentic AI programmes, the lesson is simple: if the agent can still carry dangerous values through an approved call, the identity layer is still under-governed.

Residual abuse is the named failure mode enterprise teams should now track. The article’s repeated concession is not that the patterns fail, but that they often leave a residual in the object of the action. This is the same class of problem that OWASP Agentic AI guidance describes as excessive agency and tool misuse, where the system behaves exactly as authorised and still produces harm. Practitioners should treat that residual as a first-class control gap, not an edge case.

Rigid schema boundaries are the only clean close on prompt injection, and they are rarely acceptable for assistants. The two cases that eliminate the residual do so by removing free text altogether. That is a useful security pattern for bounded tasks, but it also confirms that most real assistants remain open-output systems by design. In identity governance terms, this means the safer architecture is often narrower than the business wants, so security teams must document where open output remains an accepted risk.

Agent security is now a runtime governance problem across human, machine, and AI identity. An LLM agent is not just a model issue; it is an access issue because the agent acts through tokens, API keys, workspace permissions, and delegated trust. Where those permissions are broad, the attacker does not need to hijack the tool selection to cause damage. They only need the agent to execute a permitted action with attacker-shaped content. Practitioners should align agent design with least privilege and continuous verification, not with static prompt filters.

OWASP Agentic Applications Top 10 is the right external lens for this class of risk. The article maps cleanly to tool misuse, prompt injection, and output manipulation patterns that the broader market is still underestimating. That framing matters because it moves the discussion from model quality to control assurance. Security teams should use that lens to decide where schema, isolation, and runtime policy are mandatory, and where open-ended assistant behaviour is simply too risky.

What this signals

Residual abuse is the operational signal teams should plan for, not just prompt injection events. If an agent can still influence arguments, rankings, or summaries inside an approved action, then the monitoring problem moves to runtime behaviour and data flow rather than prompt text alone. Teams should baseline normal action objects, not just command names, and connect that view to incident triage across identity, cloud, and application control planes.

Open-output assistants will continue to create governance debt as adoption accelerates. The more the organisation relies on assistants that must write, rank, route, or summarise, the more difficult it becomes to fully eliminate injection residuals. That makes task scoping, least privilege, and behaviour-based detection the practical middle ground, especially where AI agents operate through non-human identities and delegated credentials.

Control that depends on perfect input filtering will age badly. The article’s architecture lesson is that the safer boundary is often the output schema, not the prompt filter. Teams should therefore align their agent standards with the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10, then decide where free-form generation is acceptable and where it is not.

For practitioners

• Map the residual action space Inventory every action an LLM agent is allowed to perform, then list the arguments, recipients, paths, rankings, and summaries that untrusted input can still influence. The goal is to identify where approved actions can still carry attacker-shaped values.
• Separate control flow from payload governance Use design patterns to lock tool choice and execution order, but add independent validation for the values inside those approved calls. That means checking recipient legitimacy, destination boundaries, ranking inputs, and summary sources before the action is committed.
• Constrain open-output assistants to bounded tasks Reserve rigid schemas for tasks that can tolerate them, such as classification, structured extraction, and fixed-field routing. Where the assistant must produce free text or choose among nuanced options, treat the residual as a standing governance risk rather than a defect to be patched away.
• Baseline runtime behaviour for inherited agents For agents you did not build, compare each run against the workload’s own historical behaviour so unusual endpoints, arguments, or call sequences become visible. Behavioural baselines are often the only practical way to detect residual abuse in vendor-hosted or third-party agents.

Key takeaways

• LLM agent design patterns can preserve control flow while still leaving attackable residuals in the values carried by approved actions.
• The strongest evidence in the article is that only rigid output schemas fully close the prompt injection path, which is rarely practical for general-purpose assistants.
• Security teams should govern agent payloads, runtime behaviour, and delegated identity, not just the prompt boundary.

Key terms

• Control-flow integrity: Control-flow integrity means an agent follows its intended sequence of actions even when it encounters hostile or misleading input. In LLM systems, it reduces the chance that prompt injection can redirect tool choice or execution order, but it does not automatically protect the values carried through those actions.
• Residual abuse: Residual abuse is the security gap that remains after an agent has been constrained enough to resist overt redirection. The agent still performs an authorised action, but the arguments, rankings, recipients, or outputs can be influenced by attacker-controlled content.
• Output space: Output space is the set of things an agent is allowed to emit, write, rank, summarise, or place. The broader that space is, the harder it is to eliminate prompt injection influence completely, because free-form output gives attacker input somewhere to land.
• Structured schema: A structured schema is a rigid format with fixed fields and limited values. In agent security, schemas can eliminate prompt injection influence when the task has no need for free text, but they trade flexibility for safety and are best suited to tightly bounded workflows.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

• The case-study-by-case-study breakdown of which residual survived in each agent design pattern.
• The paper's exact security grading for the ten worked examples, including the two cases where schema restriction eliminated the residual.
• The implementation trade-offs for dual LLM, plan-then-execute, map-reduce, and code-then-execute patterns in production assistants.
• The article's own explanation of how the authors distinguish control-flow integrity from output-space governance.

👉 The full ARMO post covers the ten case studies, surviving residuals, and the control-flow versus output-space distinction.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and agentic AI identity. It helps practitioners align delegated access, runtime controls, and identity lifecycle governance across modern security programmes.