LLM runtime security for AI agents is now an identity problem

At a glance

What this is: LLM runtime security is the set of live controls that monitor and constrain AI agents while they execute, with the key finding that model security alone does not cover runtime prompt, memory, tool, and output abuse.

Why it matters: It matters because IAM, PAM, and NHI teams now have to govern agent decisions, tool access, and session risk inline, not just at provisioning or launch time.

👉 Read Akto's guide to LLM runtime security for AI agents

Context

LLM runtime security is about controlling an AI agent while it is actively running, not just testing the model before deployment. That distinction matters because the security problem in AI agent identity is no longer only the model or the prompt, but the live sequence of tool calls, memory updates, and outbound actions that occur after execution starts.

Agents change the identity problem because they can read data, invoke internal APIs, and make decisions across many steps in one session. Traditional controls that assume a stable request and response pattern struggle when the actor can attach tools dynamically, carry state forward, and act on information discovered mid-session.

Key questions

Q: How should security teams govern AI agents that choose tools at runtime?

A: Security teams should treat runtime tool choice as a governed access event, not a normal application call. That means task-scoped credentials, explicit approval boundaries for sensitive actions, and logs that record both the tool selected and the identity used. If the agent can change its plan, the control model must be able to change with it.

Q: Why do AI agents complicate traditional access reviews?

A: AI agents complicate access reviews because they can accumulate permissions across tools and environments faster than manual certification cycles can observe. A review process built for stable human accounts does not fit an executor that can act across systems, create new access paths, and complete work before the next review window begins.

Q: What breaks when prompt injection reaches an AI agent's memory?

A: The attack stops being transient. Once malicious instructions are written into persistent memory, they can influence later turns, later tasks, and even separate sessions. That creates a durable control failure because the bad input survives beyond the moment it arrived and can shape future decisions.

Q: Who is accountable when an AI agent takes action through an MCP server?

A: The accountable party is the human or team that authorised the agent's access, but only if the organisation can prove that chain. Without immutable logs that connect the initiating identity to the tool call and final action, accountability becomes weak, and legal or compliance teams lose the evidence they need.

Technical breakdown

Runtime security vs model security for AI agents

Model security protects the artifact, meaning weights, training data, and alignment. Runtime security protects the running system, where prompts, retrieved documents, memory, tools, and outputs interact in real time. That distinction is critical because many abuses do not require breaking the model itself. An agent can be manipulated through indirect prompt injection, poisoned context, or a compromised tool description and still behave as designed from the model's perspective. The control problem is therefore about governing the live execution path, not just verifying the model before release.

Practical implication: Treat runtime enforcement as a separate control layer from model evaluation and red teaming.

How prompt injection, memory poisoning, and tool abuse work at runtime

Prompt injection attempts to override the agent's instructions through user input, retrieved content, or tool output. Memory poisoning is more durable because a malicious instruction or payload can persist across turns and sessions, shaping later behaviour. Tool abuse occurs when the agent is allowed to call external functions, such as email, database queries, or payment APIs, without validating whether the specific call is appropriate. Together, these attacks exploit the fact that agents mix language, state, and action. The weakness is not one bad prompt but the trust boundary between instructions and untrusted content.

Practical implication: Separate instruction channels from untrusted data and validate every tool invocation before execution.

Why MCP-connected agents need governed discovery and authorization

MCP changes the trust model because agents can discover tools and capabilities from external servers at runtime, including servers added mid-session. That means the agent is not just using predeclared tools, it is inheriting descriptions, parameters, and instructions from outside the original trust zone. If discovery is open, a compromised or misleading MCP server can expand what the agent believes it may do. Runtime authorization therefore has to check each tool call, not just the initial connection. Monitoring also needs to track server identity, tool sequences, and data passed across the boundary.

Practical implication: Govern tool discovery, tool descriptions, and per-call authorization as part of the agent's identity perimeter.

Threat narrative

Attacker objective: The attacker wants to turn the agent's legitimate runtime permissions into unauthorized business actions, data leakage, or tool-driven compromise.

1. Entry begins when an attacker injects malicious instructions through user input, retrieved documents, tool metadata, or a compromised MCP server.
2. Escalation occurs when the agent accepts those instructions into context and then uses its legitimate tool access to reach systems or data outside the intended task.
3. Impact follows when the agent carries out an unauthorized read, write, transfer, or exfiltration action that appears operationally valid from the agent's perspective.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM runtime security is an identity control problem, not just an application-layer safety problem. The article's core point is that the agent's danger emerges after deployment, when identity, memory, tools, and outputs are all active at once. That shifts the governance question from model quality to runtime authority. Practitioners should treat agent identity as a live security boundary, not a static setup step.

Dynamic tool attachment creates a runtime governance gap that static allowlists cannot close. If an agent can discover or inherit tools during execution, the original trust decision no longer describes the full access surface. This is where the named concept of runtime trust drift applies: the agent's effective privileges expand as its session evolves. Identity teams need to recognize that authorization state can change faster than review cycles can record it.

Memory poisoning breaks the assumption that risky instructions are short-lived. Traditional review logic assumes a bad input is transient and observable at the point of entry. Persistent agent memory changes that equation because malicious context can survive across turns and become future behaviour. The implication is that governance must account for durable state, not only isolated prompts.

MCP forces NHI and agentic AI governance to converge. The same questions that govern service accounts, API tokens, and privileged integrations now apply to agents that can add tools mid-session and invoke external systems through a protocol boundary. That convergence does not mean human IAM disappears; it means the enterprise now needs one identity model for human, machine, and agent behavior across the same control plane.

Runtime monitoring is the only place where abuse patterns become visible in time to matter. Session risk scoring, tool-sequence analysis, and tamper-proof audit logs are not auxiliary features here. They are the evidence layer that shows whether an agent is drifting out of scope, exfiltrating data, or chaining actions into an unauthorized outcome. Teams that cannot observe runtime behavior cannot govern agent autonomy responsibly.

From our research:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That visibility gap makes OWASP Agentic AI Top 10 a useful next reference for mapping runtime threats to concrete controls.

What this signals

Runtime trust drift: once an agent can discover tools, carry memory forward, and make sequential decisions, the real control problem becomes how far its authority can expand between approval points. That is why runtime policy engines and per-call authorization are moving from nice-to-have to mandatory control layers in AI agent programmes.

With 80% of organisations reporting that their AI agents have already performed actions beyond their intended scope, governance teams should assume that policy gaps are operational, not theoretical. The practical response is to build observability around tool use, memory writes, and session-level risk, then link those signals to identity review and incident response.

The strongest external reference point here is the OWASP Agentic AI Top 10, which aligns well with the article's runtime threat model and helps teams translate abstract agent risk into auditable control objectives.

For practitioners

• Separate instruction and data channels Treat prompts, retrieved documents, tool output, and memory as different trust classes. Do not allow untrusted content to be consumed as a command, especially when it enters the context window through RAG or tool responses.
• Require per-call authorization for every tool action Check each agent tool call at runtime against current task scope, permissions, and data sensitivity. Static pre-approval is not enough when the agent can chain multiple actions across a single session.
• Limit dynamic tool discovery Restrict which MCP servers or registries can be added to an agent session, and validate tool descriptions before they are accepted into context. Do not let discovery expand the agent's authority by default.
• Instrument session risk scoring Score injection attempts, unusual tool sequences, sensitive data access, and policy near-misses together so the platform can tighten controls or terminate the session before damage compounds.
• Log agent activity as identity evidence Keep tamper-proof audit logs that tie the agent identity, the policy decision, the tool call, and the resulting action together. That is what enables incident reconstruction and access review after the fact.

Key takeaways

• LLM runtime security is about governing what an AI agent does while running, which makes identity, memory, and tool use part of the attack surface.
• The clearest evidence of risk is runtime behaviour, including prompt injection, unauthorized tool calls, and session-level scope drift.
• Practitioners need per-call authorization, governed tool discovery, and runtime auditability before agentic systems can be trusted in production.

Key terms

• Runtime LLM Security: Runtime LLM security is the set of controls that observe and enforce policy while an AI model is actively in use. It focuses on prompts, outputs, tool calls, and access to external data so defenders can block misuse during live operation, not only during pre-deployment testing.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• MCP: Model Context Protocol, an open way for AI agents to connect to tools and data sources. It improves interoperability, but it also introduces a shared integration layer that must be governed carefully because the protocol can widen access across many systems at once.
• Access Risk Scoring: Access risk scoring assigns a risk value to users, accounts, or entitlements based on privilege, behaviour, and policy context. It helps teams prioritise remediation, but only if the score triggers an actual change to access rather than remaining a reporting metric.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step runtime control architecture for prompts, memory, tools, monitoring, and outputs
• Detailed explanation of MCP-specific discovery and authorization risks in production
• Examples of runtime policy enforcement, session risk scoring, and human approval workflows
• The article's full FAQ covering prompt injection, tool validation, and agent observability

👉 Akto's full post covers the runtime control layers, threat patterns, and production monitoring detail

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.