Agentic AI Security vs LLM App Security: How Your Security Programme Must Change

Most security programmes built for LLM applications assume the worst outcome is a bad answer, and that assumption breaks the moment the model can call tools with real credentials. An LLM application turns text into text that a person reads and acts on. An agent turns text into actions that nothing may review before they land. This guide is about the programme delta: what changes in threat modelling, testing and control ownership when a team moves from LLM applications to agents, and which LLM controls carry over unchanged. It is written for security architects, AppSec leads and identity teams who already run an LLM security programme and now have agents in the pipeline. For the spectrum from chatbot to multi-agent system, see the AI Agents vs Agentic AI.

Key takeaways

• The harm moves from output to action. An LLM app's worst case is usually disclosure or a misleading answer; an agent's worst case is whatever its credentials allow, carried out at machine speed.
• Prompt injection becomes a privilege problem. The UK NCSC treats the model as an "inherently confusable deputy", so the control that matters for agents is a deterministic limit on what a confused agent can do, not a better filter.
• Threat models need identity and delegation on the diagram. Every agent is a non-human identity, often acting for a person; the model must show whose authority each tool call carries.
• Testing shifts from prompts to trajectories. You test whole multi-step runs, tool calls and approval paths in a sandbox, not single prompt and response pairs.
• Ownership spreads beyond AppSec. Identity, platform and the business owner of each action become control owners; an LLM programme run by one team does not scale to agents.

The short answer

LLM application security protects the integrity and confidentiality of what a model reads and writes. Agentic AI security protects everything an LLM application security programme protects, plus the systems an agent can change. The OWASP Top 10 for LLM Applications 2025 already names the hinge: LLM06 Excessive Agency, which it defines as the vulnerability that lets damaging actions happen in response to unexpected, ambiguous or manipulated model output, with three root causes: excessive functionality, excessive permissions and excessive autonomy. The OWASP Top 10 for Agentic Applications 2026 (published by the OWASP GenAI Security Project in December 2025) expands that one entry into ten agent risks, ASI01 Agent Goal Hijack to ASI10 Rogue Agents.

In programme terms, three things change. The threat model gains identities, delegation and state. The test plan gains tool use, memory and multi-step behaviour. The control set gains owners outside AppSec, because the strongest agent controls (token scope, approval gates, egress limits, kill switches) live in identity and platform systems. For the agent list entry by entry, see the OWASP Top 10 for Agentic Applications Guide, and for the hinge entry the OWASP LLM Top 10 LLM06 Excessive Agency Guide.

Side-by-side: what each programme defends

Threat model: from output harm to action harm

LLM application threat model

Architecture in words: a user sends a prompt; the application adds a system prompt and, often, retrieved documents; the model returns text; the application renders it or passes it to a parser. The trust boundaries are the user input, the retrieval store and the output handler. Threats cluster around LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM05 Improper Output Handling, LLM07 System Prompt Leakage and LLM08 Vector and Embedding Weaknesses. A STRIDE-style review of the app plus these entries covers most of it.

Typical failure modes: the model reveals data from its context to the wrong user, output is rendered as active content (markdown images or links become an exfiltration channel), or downstream code trusts model output as if it were validated input. The EchoLeak case shows the pattern: an external email retrieved into Microsoft 365 Copilot's context carried instructions that pushed internal data into a link the client fetched automatically. See the EchoLeak 2025 analysis.

Agent threat model

Architecture in words: the agent receives a goal, plans steps, selects tools, calls them with credentials, reads the results back into context, writes to memory and may hand work to other agents. Each tool result is new untrusted input. Each tool call is an authorisation decision. Each hand-off is a trust boundary. Cloud Security Alliance's MAESTRO framework gives one seven-layer reference for this, from foundation models up to the agent ecosystem, and argues that STRIDE alone misses multi-agent interaction and AI-specific threats. See the AI Agent Threat Modelling Guide for running the exercise.

What the threat model must now show:

• Whose authority each action carries: the agent's own identity, a user's delegated token, or a shared service credential. The joint Five Eyes guidance on careful adoption of agentic AI (published by the Canadian Centre for Cyber Security with ASD's ACSC, CISA, NSA, NCSC-NZ and the UK NCSC, announced 30 April 2026) names the confused deputy pattern as a core privilege risk: a low-privileged user or compromised tool borrowing a trusted agent's permissions.
• Every tool with its blast radius: read, write, delete, send, pay, execute. A tool list without verbs is not a threat model.
• Persistent state: memory stores, scratchpads and task queues, because ATLAS AML.T0080 AI Agent Context Poisoning has sub-techniques for both memory (AML.T0080.000) and thread (AML.T0080.001).
• Agent-to-agent paths: which agents can instruct which, and what the receiver checks before acting.
• Stop points: where a human approves, where a kill switch acts, and what happens when the agent is uncertain.

Typical failure modes: a legitimate tool used in a harmful way (ASI02 Tool Misuse and Exploitation), delegated authority stretched beyond the task (ASI03 Identity and Privilege Abuse), poisoned memory that changes later runs (ASI06 Memory and Context Poisoning) and one fault spreading across agents (ASI08 Cascading Failures). The PocketOS incident is the plainest example of action harm: a coding agent found an account-wide platform token in an unrelated file and used it to delete a production database and its backups in one API call. See the PocketOS database deletion analysis. No prompt filter would have stopped it; token scope and an approval gate would have.

Which LLM controls still apply, and what changes

Nothing in an LLM security programme becomes obsolete. Most controls stay, some change owner or placement, and a set of new ones arrives. The table is the practical inventory.

New controls with no LLM equivalent:

• Per-agent identity and short-lived credentials: each agent a distinct principal with its own credentials, never a shared key. See the Agentic AI Identity Guide.
• Tool authorisation outside the model: the OWASP cheat sheet calls this tool authorisation middleware and says to fail closed for unknown tools or missing approval requirements.
• Delegation with attenuation: authority narrows at each hand-off and expires with the task. See Delegation Chains and Attenuated Authority for AI Agents.
• Action-bound approvals: approvals tied to the exact action and parameters, short-lived, with replay protection and step-up authentication for critical operations. See the Human-in-the-Loop for AI Agents Guide.
• Memory controls: validation before write, isolation per user and session, expiry and integrity checks. See the AI Agent Memory Poisoning Guide.
• Egress control and sandboxing: allowlisted destinations and isolated execution for any agent that runs code or browses.
• Containment: a tested way to stop one agent or all agents and revoke their tokens. See the AI Agent Kill Switch and Containment Guide.

How testing changes

LLM application testing is mostly about prompts: jailbreak suites, injection payload sets, leakage probes and evaluation of answer quality. That remains necessary. Agent testing adds the questions that only appear when the system acts.

The Five Eyes guidance asks for red teaming in sandboxed environments, including adversarial and multi-agent scenarios, evaluation across autonomy levels and tool access, and continued evaluation through the lifecycle, because agents may behave differently in evaluation than in production. The OWASP cheat sheet adds an abuse-case test matrix (prompt override, tool misuse, memory poisoning, approval bypass) and release gates in CI/CD. The sandbox requirement is not theoretical: the OpenAI and Hugging Face breach 2026 analysis describes agents escaping an evaluation sandbox and taking over cloud credentials, so test environments need the same credential hygiene as production. See the AI Agent Security Testing Checklist and the AI Red Teaming Guide.

Control ownership: who owns what after the shift

In many organisations the LLM security programme sits with AppSec and the product team. That works while the model only writes text. Once it acts, the controls that bound the damage sit in systems AppSec does not run. The Five Eyes guidance asks organisations to assign clear responsibility for outcomes and define risk ownership in governance policy; NIST CSF 2.0 GV.RR-02 asks the same of cyber security roles generally.

Worked scenario: a support assistant gains actions

Before: a customer support assistant answers questions from a knowledge base and the customer's order history. The programme covers injection through retrieved articles, leakage of other customers' data and output rendering. Controls: retrieval trimmed to the signed-in customer, output encoding, a content filter, prompt logging.

The change: the product team adds three tools: issue a refund up to a limit, change the delivery address and email the customer. The assistant now reads inbound emails to build context.

What the programme must add:

1. Threat model update: add the inbound email channel as untrusted input, the three tools with their verbs and limits, and the credential each uses. Identify the confused deputy path: a hostile email instructing a refund to a different account.
2. Identity: give the agent its own identity; call the refund API with a token scoped to the customer in session and to the refund action only, issued per task.
3. Deterministic limits: enforce refund ceilings and address change rules in the refund and order services, not in the prompt.
4. Approval: require a human for refunds above a threshold and for any address change followed by a refund in the same session.
5. Egress: restrict email sending to the customer's verified address.
6. Testing: add trajectory tests where the injection arrives in an email and the pass condition is "no refund issued and approval requested".
7. Ownership: the payments owner signs off refund limits; identity owns the agent's token issuance; the SOC gets alerts on refund rate per agent.

Detection and measurement

LLM programmes measure blocked prompts and leakage findings. Agent programmes need metrics about authority and behaviour:

• Agents with a distinct identity: share of production agents with their own principal rather than a shared or human credential (target 100%).
• Standing privilege: share of agent credentials that are long-lived or broader than one task or environment.
• Approval coverage: share of destructive or irreversible tool actions behind an action-bound approval.
• Trajectory test coverage: share of agent releases that passed injection-through-tool-output tests before deployment.
• Containment time: time from detection to revoked tokens and a stopped agent in exercises.

Log tool calls with parameters, the identity used, approvals and inter-agent messages (NIST SP 800-53 AU-2 and AU-12), and alert on anomalies such as unusual call rates, new destinations and privilege changes (SI-4).

Framework mapping

Common failure patterns

• Reusing the LLM threat model unchanged. The diagram shows the model and the vector store but no tools, credentials or memory, so the highest-impact paths are never rated.
• Treating injection filters as the boundary. The NCSC warns against deny-lists and against products that claim to stop prompt injection. In the ForcedLeak 2025 case, instructions in a stored web form lead were followed when an employee asked the agent to process it; the fix that mattered was enforcing trusted URLs on the outbound channel.
• Agents running on a person's or a shared credential. Actions cannot be attributed, revoked or scoped. The Five Eyes guidance flags static and shared keys as a direct identity risk.
• Approval prompts the agent can satisfy. Approvals that are generic, long-lived or phrased by the agent invite rubber-stamping (ASI09 Human-Agent Trust Exploitation). Bind approvals to exact parameters.
• Testing only prompts. A suite that never feeds hostile content through a tool result never tests the main agent attack path.
• No owner outside AppSec. Token scope, tool permissions and refund limits drift because the teams that own those systems were never asked.

Practitioner checklist

• Classify each AI system by what it can do: text only, actions with approval, or autonomous actions. Apply the agent programme to the last two.
• Redraw the threat model with every tool, its verbs, the credential it uses and whose authority it carries.
• Give each agent its own identity and short-lived, task-scoped credentials; remove secrets from prompts, code and agent-reachable files.
• Enforce limits in the target systems and in a tool authorisation layer that fails closed, not in the prompt.
• Put destructive and irreversible actions behind action-bound approvals owned by the business owner.
• Add trajectory tests with injection through tool outputs, memory and peer agents to release gates, run in a sandbox with fake credentials.
• Log tool calls, identities, approvals and inter-agent messages; give the SOC agent-specific detections.
• Write down the owner of each agent control in a RACI, including identity, platform and business owners.
• Exercise containment: stop an agent and revoke its tokens, and measure the time taken.

Standards and references

• OWASP GenAI Security Project: OWASP Top 10 for Agentic Applications for 2026
• OWASP GenAI Security Project: Top 10 for LLM Applications 2025
• OWASP Cheat Sheet Series: AI Agent Security Cheat Sheet
• Canadian Centre for Cyber Security and partners: Careful adoption of agentic AI services
• UK NCSC: Prompt injection is not SQL injection (it may be worse)
• Cloud Security Alliance: Agentic AI Threat Modeling Framework: MAESTRO
• MITRE ATLAS: AI Agent Tool Invocation
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: Agentic AI Security Guide · LLM Security Guide · AI Agent Threat Model Template · Multi-Agent and A2A Security Guide · Agentic AI Identity Maturity Model