LLM app security is shifting from leakage to manipulation risk

At a glance

What this is: This is an analysis of how LLM application security is shifting from leakage concerns to manipulation risk, with system prompts becoming part of the security boundary.

Why it matters: IAM, AppSec, and AI governance teams need to treat prompts, guardrails, and runtime behaviour as controlled surfaces because attackers can change what the application does, not just what it reveals.

Context

Traditional application security assumes deterministic behaviour, where a known input should produce a predictable outcome. LLM-based applications break that assumption because their outputs and internal decision paths can shift in response to prompt content, model context, and external inputs.

That creates a governance problem for AI-enabled application programmes: the security boundary is no longer just code and configuration, but also system prompts, guardrails, and runtime behaviour. For identity and access teams, this is closest to controlling an operational control plane than protecting static application text.

The article’s core point is that leakage is still real, but manipulation is the more structurally important risk for LLM applications and agents.

Key questions

Q: What breaks when agent prompts are treated like ordinary text?

A: When prompts are treated as ordinary text, the system has no reliable way to distinguish authorised directives from injected, replayed, or modified instructions. That weakens both integrity and accountability, because the agent can act on content that was never meant to carry execution authority. Signed directives restore a verifiable trust boundary before action begins.

Q: Why do LLM applications create new data leakage risks for identity teams?

A: LLM applications can expose sensitive data when users paste secrets, when agents retrieve privileged context, or when responses echo internal material back to users. That creates an identity problem because the model may handle information it was never meant to disclose. Teams should govern what the model can see and what it can return.

Q: What are the signs that guardrails are failing in an LLM application?

A: Guardrails are likely failing when the model returns unsafe, off topic, or unstructured responses that bypass policy checks. Other warning signs include inconsistent formatting, missing validation of sensitive outputs, and corrective actions that do not trigger when criteria are not met. If users can still provoke harmful or noncompliant replies, the control is too weak or misconfigured.

Q: How should security teams govern AI prompts that include sensitive data?

A: Treat the browser as a control point, not just an interface. Inspect the sensitivity of the data, the identity of the user, and the context of the session before the prompt leaves enterprise control. That lets teams allow useful AI use while blocking risky disclosure paths without relying only on after-the-fact DLP.

Technical breakdown

Why system prompts behave like a control plane

System prompts are not ordinary content. They define behaviour, tone, guardrails, and task scope for an LLM application, which means they sit closer to policy than to user-facing text. If a malicious input can override or redirect that prompt, the application can disclose protected context, accept unsafe instructions, or change how it makes decisions. In AppSec terms, the prompt is part of the executable boundary even when it is written in natural language. That makes prompt review, constraint design, and change control central to securing the application.

Practical implication: treat system prompts as governed security artefacts, not editable prose.

Prompt injection and model manipulation in LLM apps

Prompt injection occurs when an attacker supplies text that influences the model to ignore prior instructions or follow new ones. In an LLM application, that can mean exposing hidden instructions, altering workflow logic, or steering the model into unsafe outputs. The article also points to model drifting, where behaviour changes over time if prompts, context, or inputs are not tightly managed. Together, these are non-deterministic failure modes, which means conventional test cases will miss some of the highest-risk edge conditions.

Practical implication: test for instruction override, behavioural drift, and unsafe output paths rather than only for data exposure.

Why resource abuse matters in LLM security

LLM-based applications can be manipulated into expensive or endless execution patterns, including denial-of-service style loops and wallet abuse. That risk emerges because the model can be pushed into repeated reasoning, oversized context handling, or repeated tool-like operations without the clear termination logic expected in classic software. When cost, availability, and behaviour are coupled in one runtime, abuse is both a security and operational problem. This is why LLM AppSec has to account for usage boundaries, not only content filters.

Practical implication: enforce runtime limits and abuse detection around token, loop, and call-volume behaviour.

Threat narrative

Attacker objective: The attacker wants to manipulate the application’s behaviour so the model reveals protected information, bypasses intended controls, or burns resources.

1. Entry begins when a malicious user input or external content reaches the LLM application and is interpreted alongside the system prompt.
2. Credential or control compromise occurs when prompt injection changes the model’s instruction hierarchy and causes it to follow attacker-supplied directions instead of intended guardrails.
3. Escalation follows as the model reveals hidden configuration, alters transaction logic, or enters repeated high-cost execution patterns.
4. Impact is manipulation of application behaviour, operational disruption, or unintended disclosure of sensitive instructions and configuration.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM application security is no longer a leakage-only problem. The industry’s early focus on accidental data exposure was too narrow for the runtime reality of generative AI. Once the model can be steered through prompt content, the security question becomes whether the application can be manipulated at all. Practitioners should now treat manipulation risk as the primary AppSec problem for LLM-based systems.

System prompts have become policy objects, not implementation notes. Traditional AppSec separates code from content, but LLM applications blur that line because the prompt defines operational behaviour. That means review, approval, and change governance need to apply to prompts with the same seriousness once reserved for code and access policy. The practical implication is that prompt governance must move into the application security model itself.

Non-deterministic behaviour breaks test assumptions that conventional AppSec depends on. Deterministic software can be validated against expected paths, but LLM systems can produce different outcomes from the same prompt depending on context and surrounding inputs. That invalidates static-only assurance approaches and makes runtime monitoring a structural requirement. Security teams should expect control gaps to appear at the interaction layer, not only in source code.

Model drifting creates an operational trust problem, not just a quality problem. If prompts, context, or inputs are not tightly monitored, an LLM application can gradually move away from approved behaviour without a clear deployment event. That is a governance failure because the control surface is live and mutable. Practitioners need to recognise that behaviour assurance for LLMs is continuous, not one-time.

Prompt injection should be analysed as a boundary-crossing attack on application intent. The attack matters because it lets external text compete with internal instructions for control over the model’s next action. That is why the right question is not whether the model can answer correctly in normal conditions, but whether it can be coerced into violating its intended operating rules. Teams should build their AppSec programme around that failure mode.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Prompt governance is becoming part of application security, not a side concern. Teams that separate AI behaviour from the rest of the AppSec programme will miss the fact that prompts now shape runtime control, approval logic, and user-facing outcomes. The governance model has to move from code-only assurance to behaviour assurance at the interaction layer.

LLM system prompts are now a live control surface. That means the programme question changes from “is the model accurate?” to “can the model be coerced into unsafe action?” For practitioners, this shifts review effort toward prompt versioning, behavioural testing, and runtime monitoring rather than static content checks alone.

For practitioners

• Define prompts as governed security artefacts Put system prompts under version control, peer review, and formal approval so changes are treated as security-relevant configuration changes.
• Test for instruction override paths Create adversarial test cases that try to override system instructions, extract hidden context, and force unsafe behavioural changes.
• Set runtime limits on model execution Constrain token budgets, recursion-like loops, and repeated tool-triggered actions to reduce denial-of-service and wallet abuse risk.
• Add continuous behavioural monitoring Watch for drift in output style, decision quality, policy adherence, and unusual resource consumption after deployment.
• Align AppSec, AI, and product teams Assign shared ownership for prompt design, guardrails, abuse testing, and incident escalation so security is built into the lifecycle.

Key takeaways

• LLM applications now create manipulation risk, not just leakage risk, because prompt content can alter model behaviour at runtime.
• System prompts function as part of the control plane, which means prompt changes need governance, review, and monitoring.
• Security teams should build behavioural testing and runtime oversight into AppSec programmes for AI-enabled applications.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• System Prompt: The system prompt is the core instruction set that defines an LLM’s behaviour, boundaries, and response style during a session. When attackers influence or override it, they are not merely changing text. They are trying to change the model’s governing authority.
• Model Drift: Model drift is the gradual change in a model’s behaviour or performance after deployment. It happens when the operating environment, user patterns, or inputs no longer match the conditions used to validate the system. Drift matters because a model can appear functional while no longer meeting approved standards.
• Behaviour Assurance: Behaviour assurance is the practice of validating what an AI system actually does in production, not just what it should do on paper. For LLM applications, it means monitoring outputs, guardrails, and runtime patterns continuously because non-deterministic systems can change state without a traditional release event.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.