LLM Security Guide: Jailbreaks, Data Leakage and Outputs

A large language model application is an untrusted, probabilistic component wired into trusted systems, and most LLM incidents happen because teams forget that on one side of the model or the other. Attackers talk the model out of its safety behaviour, extract data it should never have seen, push its output into browsers, shells and databases, and run up the inference bill. This guide covers jailbreaks, data leakage and system prompt extraction, output handling, model and endpoint security, and how the OWASP Top 10 for LLM Applications 2025 and MITRE ATLAS describe them. Prompt injection is summarised and linked to its own guide. It is written for application security engineers, AI platform teams and architects who build or approve LLM applications, and it contains no working attack prompts.

Key takeaways

Treat the model as an untrusted user on both sides. Inputs can steer it and outputs can carry payloads, so validation, encoding and authorisation belong in deterministic code around the model, not in the prompt.
Jailbreaks are a when, not an if. Many-shot, multi-turn and format-based jailbreaks have transferred across every major model family tested; plan for the model's safety training to fail and limit what a jailbroken model can reach.
The system prompt is not a secret or a control. OWASP LLM07 is explicit: keep credentials, roles and business limits out of prompts and enforce them in code.
Data leakage is mostly an access control problem. A model can only disclose what it was trained on, retrieved or handed in context, so scope data sources per user before inference.
Inference endpoints are high-value infrastructure. Rate limits, quotas, authentication, logprob restrictions and key hygiene stop denial of wallet, model extraction and LLMjacking.

The LLM application attack surface

An LLM application has more trust boundaries than a typical web app, each with its own failure mode.

Layer	What attackers target	OWASP LLM 2025	MITRE ATLAS
User prompt	Override application intent, bypass safety training	LLM01 Prompt Injection	AML.T0051, AML.T0054 LLM Jailbreak, AML.T0065 LLM Prompt Crafting
Context (system prompt, RAG, memory)	Extract instructions, retrieve other users' data, poison retrieval	LLM02, LLM07, LLM08 Vector and Embedding Weaknesses	AML.T0056 Extract LLM System Prompt, AML.T0057 LLM Data Leakage, AML.T0071 False RAG Entry Injection
Model output	XSS, SQL injection, code execution, data exfiltration through rendering	LLM05 Improper Output Handling, LLM09 Misinformation	AML.T0050 Command and Scripting Interpreter, AML.T0077 LLM Response Rendering
Tools and plugins	Use the model's permissions as a confused deputy	LLM06 Excessive Agency	AML.T0053 AI Agent Tool Invocation
Inference endpoint	Denial of service, denial of wallet, model extraction, stolen keys	LLM10 Unbounded Consumption	AML.T0029 Denial of AI Service, AML.T0034 Cost Harvesting, AML.T0024 Exfiltration via AI Inference API, AML.T0055 Unsecured Credentials
Model and data supply chain	Backdoored weights, malicious adapters, poisoned fine-tuning data	LLM03 Supply Chain, LLM04 Data and Model Poisoning	AML.T0010 AI Supply Chain Compromise, AML.T0058 Publish Poisoned Models, AML.T0020 Poison Training Data

ATLAS IDs are from the ATLAS 5.6.0 data file; check them against the current matrix when mapping detections.

Prompt injection in brief

Prompt injection (OWASP LLM01) underlies many risks below: content the model reads, typed by the user (direct) or embedded in a web page, email, document or tool result (indirect), changes what the model does. Inside the model there is no reliable boundary between instructions and data, so injection is a residual risk. The controls that hold are outside the model: least privilege for tools, human approval for consequential actions, egress restrictions and output handling. See the Prompt Injection Guide for channels, hiding techniques and design patterns. The EchoLeak zero-click flaw in Microsoft 365 Copilot shows indirect injection and output rendering combining into data exfiltration.

Jailbreaks: how they work and how to contain them

A jailbreak targets the model provider's safety training rather than your application's logic. ATLAS tracks it as AML.T0054 LLM Jailbreak under Privilege Escalation and Defense Evasion; Microsoft notes that jailbreaks are also called direct prompt injection attacks. Microsoft classifies Skeleton Key as "Explicit: forced instruction-following" by someone with legitimate access: for enterprise apps jailbreaks are mostly an authenticated-user problem, for consumer apps a public abuse problem.

Technique family	Mechanism (high level)	What the research showed	Containment
Many-shot jailbreaking	A single long prompt containing a fabricated dialogue of many compliant answers, then the real request	Anthropic (April 2024) tested up to 256 shots; success followed a power law in shot count, and larger models were often more susceptible	Classify and modify prompts before inference (one method cut success "from 61% to 2%"); cap context length per use case
Multi-turn escalation (Crescendo)	Starts with a benign question and escalates gradually, building on the model's own replies	Russinovich et al. reported high success across ChatGPT, Gemini, LLaMA and Anthropic models, and automated it	Score the whole conversation, not only the last turn; reset or end sessions on escalation
Guideline augmentation (Skeleton Key)	Persuades the model to "update" its rules and add a warning instead of refusing	Microsoft found it worked on most models tested in April and May 2024	Input filtering, system message, output filtering and abuse monitoring as separate layers
Format and role play (Policy Puppetry)	Prompt dressed as an XML, INI or JSON policy file, combined with role play and leetspeak	HiddenLayer (April 2025) reported one template transferring across major frontier and open models, and adapted it for system prompt extraction	Do not rely on RLHF alone; output-side classifiers; red team with format variants
Obfuscation and encoding	Base64, other languages, character substitution to slip past keyword filters	Keyword and deny-list filters miss encoded or translated requests	Classifiers on decoded text; avoid deny-lists
Fine-tuning circumvention	Custom fine-tuning removes safety behaviour from an aligned model	Qi et al. (2023) broke GPT-3.5 Turbo guardrails with 10 examples for under $0.20; benign fine-tuning also degraded safety	Re-run safety evaluations after every fine-tune; restrict who can fine-tune and on what data

Layer, do not trust a single filter: Microsoft's stack for Skeleton Key is input filtering, a system message, output filtering and separate abuse monitoring; none is sufficient alone.
Classify outputs as well as inputs: obfuscated prompts are hard to classify; the harmful output usually is not.
Decide what a jailbreak can actually reach: a jailbroken chatbot with no tools and no private data produces embarrassing text. The DPD chatbot incident and the Chevrolet dealer chatbot $1 Tahoe were reputational. The same jailbreak against an agent with write access is an incident.

Data leakage and system prompt extraction

OWASP LLM02:2025 Sensitive Information Disclosure covers personal data, credentials, business data and proprietary logic exposed through output. Each of the four sources needs a different control.

Source of leaked data	Example	Primary control
Training or fine-tuning data	Model reproduces memorised records or secrets	Scrub and minimise training data, differential privacy where feasible, extraction testing before release (AML.T0024.000)
Retrieved context (RAG)	Another user's documents appear in an answer	Permission-aware retrieval that filters by the caller's identity before the model sees content
System prompt and hidden instructions	Credentials, internal limits or role names extracted (AML.T0056)	Nothing sensitive in prompts; enforce rules in code
User input sent to the provider	Staff paste source code or customer data into a public tool	Approved tools with no-training terms, input DLP, logging and retention limits

System prompt leakage (LLM07): OWASP says disclosure of the prompt itself "does not present the real risk"; the risk is an application that delegates "strong session management and authorization checks" to the model. Attackers can often infer guardrails from behaviour anyway, so assume the prompt is public.
Prompt-level restrictions are weak: OWASP warns that system prompt restrictions on what data to return "may not always be honored and could be bypassed via prompt injection or other methods". Redact or tokenise sensitive fields before they reach the context window.
Cross-user leakage: shared caches, vector indexes and conversation history keyed only by session are common causes. Partition by tenant and user, and test with two accounts. See the Permission-Aware RAG Guide.
Shadow use: the Samsung ChatGPT source code leak began with staff pasting code into a public tool; offer an approved alternative, not just a ban. See the AI and Privacy Guide for training data and personal data obligations.
Logs are a leakage store: prompt and completion logs need the classification and retention of the source data; the DeepSeek database exposure leaked chat logs and API secrets from an open database.

Output handling: never execute what the model says

OWASP LLM05:2025 Improper Output Handling is "insufficient validation, sanitization, and handling of the outputs" before they reach downstream components, leading to XSS, CSRF, SSRF, privilege escalation or remote code execution. Whoever influences the prompt reaches whatever consumes the output.

Output sink	Failure	Control
Browser (chat UI, generated HTML or Markdown)	XSS; Markdown images that send data to an external URL	Context-aware output encoding, strict Content Security Policy (img-src and connect-src allowlists), disable remote image rendering
Database	Generated SQL runs with the app's privileges	Parameterised queries or prepared statements, read-only database role, allowlisted query templates
Shell, eval, code interpreter	Remote code execution	No exec or eval on output; sandbox with no network, no secrets, seccomp and setrlimit limits
File system	Path traversal from generated file names	Canonicalise paths, fixed base directory, allowlisted extensions
Package installs from generated code	Hallucinated package names registered by attackers	Install only from an internal mirror or allowlist; verify package existence and age
Other tools and plugins	Cross plugin request forgery	No plugin-to-plugin calls by default; authenticated user confirmation

Remote code execution is real: the LLMSmith study (Liu et al., ACM CCS 2024) tested 6 frameworks and 51 apps, found 13 vulnerabilities (7 CVEs scored 9.8) and 16 apps vulnerable to remote code execution, and recommends least privilege and environment isolation. The Langflow CVE-2025-3248 exploitation shows what happens when an AI workflow server executes code reachable from the network.
Confused deputy through plugins: Johann Rehberger's cross plugin request forgery (May 2023) used a browsing plugin to load a page whose hidden instructions made ChatGPT call an email plugin and send a summary to an attacker's URL. Zapier fixed it with an authenticated confirmation step; an unauthenticated confirmation link can itself be clicked by the model.
Misinformation is an output risk too: LLM09 covers confident but wrong answers; the Air Canada chatbot ruling made the airline liable for its bot's answer. Ground policy and price answers in approved sources.

Model and endpoint security

OWASP LLM10:2025 Unbounded Consumption covers an application that "allows users to conduct excessive and uncontrolled inferences": variable-length input floods, denial of wallet, continuous input overflow that keeps hitting the context window limit, resource-heavy queries, model extraction through the API and functional replication by training on outputs. The matching ATLAS techniques are AML.T0029 Denial of AI Service, AML.T0034 Cost Harvesting and AML.T0024.002 Extract AI Model.

Bound every request and caller: maximum input and output tokens, timeouts, caps on queued actions, and per-user and per-tenant rate limits and spend caps, with provider budget alerts as a second line.
Reduce what the API reveals: OWASP recommends restricting or obfuscating logprobs and logit_bias, which help extraction attacks. Expose them only to callers who need them.
Authenticate inference servers: self-hosted inference and orchestration servers (vLLM, Ollama, Ray, Langflow and similar) must not listen on public interfaces without authentication. The ShadowRay attacks on exposed Ray clusters stole cloud credentials and AI tokens from unauthenticated AI infrastructure, and the McKinsey Lilli incident started with unauthenticated APIs.
Secure the model artefact: OWASP LLM03 warns that model cards "offer no guarantees on the origin of the model" and that malicious LoRA adapters can compromise a base model. Verify signatures and hashes, prefer weight formats that do not execute code on load, and record models and adapters in an AI bill of materials. See the AI Supply Chain and AI-BOM Guide.
Protect fine-tuning pipelines: restrict write access to training data and model registries, and re-run safety and leakage evaluations after each training run.

Where identity fits

Provider API keys, gateway master keys, retrieval service accounts and plugin OAuth tokens are non-human identities with direct cost and data impact (ATLAS AML.T0055 Unsecured Credentials). Stolen keys become free inference for attackers, as the LLMjacking campaigns and Storm-2139 abuse of Azure OpenAI showed. Keep keys in a secrets manager, scope them per application and environment, prefer workload identity federation, and give retrieval connectors the caller's identity, not a broad service account. See the LLMjacking Guide and the API Key Management Guide.

Choosing controls for your deployment

Decision	Option A	Option B	Choose A when
Model hosting	Provider API (managed)	Self-hosted open-weight model	The provider's data terms fit your classification; choose B for strict residency or offline use, accepting that you then own patching, authentication and abuse monitoring
Guardrails	Provider content filters	Separate guardrail service or classifier you control	Single provider, low risk; choose B for several models or consistent policy and logging
Data access	Context passed per request, filtered by the caller's permissions	Fine-tuning on internal data	Data has per-user access rules (most enterprise data); fine-tune only on data every user may see
Output use	Human reads the output	Output drives code, queries or actions	Always possible; B needs output schemas, validation, sandboxing and approval gates first

Small organisations: a managed provider with enterprise data terms, its content filters and spend limits on, read-only use cases, and a proxy for logging and keys.
Large organisations: a central LLM gateway that owns provider keys, per-team quotas, guardrail policy, logging and model allowlists, so teams do not each rebuild them.

Implementing LLM security in phases

Inventory: list every LLM application, model, provider, key, data source and tool, including shadow use seen in proxy and expense data.
Threat model each application: map it to OWASP 2025 and ATLAS, and record what a fully jailbroken model could reach.
Fix the deterministic layer first: move secrets out of prompts, enforce authorisation in code, apply output encoding, parameterised queries and CSP, and remove exec paths.
Bound the endpoint: token limits, timeouts, per-identity quotas, spend alerts, authentication on every inference and orchestration server.
Add guardrails in monitor mode: run input and output classifiers in log-only mode for two to four weeks, measure false positives on real traffic, then enforce on high-risk categories.
Red team before release and after change: test jailbreaks, extraction, cross-user leakage and output injection with tooling such as PyRIT, and re-run after model, fine-tune and prompt changes. See Red Teaming AI Agents for Identity Abuse.
Roll out in rings: internal pilot, limited group, then general availability, behind a feature flag that disables tools or reverts the model and prompt. Version prompts and guardrail policies like code so rollback is a deployment.

Detection and measurement

Log each request with caller identity, application, model and version, prompt template version, token counts, guardrail verdicts, tools called with arguments, and output destination; retain full prompt text only where retention rules allow. The NCSC guidelines for secure AI system development put logging and monitoring in the operation and maintenance stage.

Alert on: spikes in tokens or spend per identity, many long prompts near the context limit, repeated refusals or guardrail hits from one account, requests for logprobs at volume, outputs containing URLs to unknown domains, and API key use from new networks.
Guardrail block rate and false positive rate: measured per category; a falling block rate with stable traffic can mean a new bypass.
Red team attack success rate: per jailbreak family and per release, with denominators, tracked over time.
Applications with secrets in prompts: target zero, checked by scanning prompt templates in source control.
Endpoints without authentication or quotas: target zero across self-hosted and gateway routes.
Mean time to revoke a leaked LLM API key: from detection to revocation and replacement.

Framework mapping

Control area	OWASP LLM 2025	NIST SP 800-53 Rev. 5	CIS Controls v8.1	ISO/IEC 27001:2022 Annex A
Input validation and guardrails	LLM01	SI-10 Information Input Validation	16 Application Software Security	A.8.28 Secure coding
Output filtering and encoding	LLM05, LLM02	SI-15 Information Output Filtering	16	A.8.28
Least privilege for tools and data	LLM06, LLM07	AC-6 Least Privilege	6 Access Control Management	A.8.2 Privileged access rights, A.5.15 Access control
Endpoint limits	LLM10	SC-5 Denial-of-Service Protection	No direct safeguard	A.8.6 Capacity management
Model and data supply chain	LLM03, LLM04	SR-3 Supply Chain Controls and Processes	16	A.5.21 Managing information security in the ICT supply chain
Logging and monitoring	LLM10	AU-2 Event Logging, SI-4 System Monitoring	8 Audit Log Management	A.8.15 Logging, A.8.16 Monitoring activities
Adversarial testing	All	CA-8 Penetration Testing	18 Penetration Testing	A.8.29 Security testing in development and acceptance

NIST AI 600-1 lists Information Security, Data Privacy, Confabulation and Value Chain and Component Integration among its 12 generative AI risks, and suggests red-teaming against prompt injection and data poisoning (MS-2.7-007).

Common failure patterns

Authorisation in the system prompt: "only discuss the user's own account" written as an instruction while retrieval can read every account; one jailbreak and the rule is gone.
Rendering untrusted Markdown: the chat UI loads remote images and links from model output, giving any injected instruction an exfiltration channel, as in EchoLeak.
Agent tools with the developer's privileges: a code interpreter or SQL tool running with write access, no sandbox and outbound network, which the LLMSmith study found exploitable to remote code execution.
Unauthenticated AI infrastructure: inference, notebook and orchestration servers exposed to the internet by default, as in ShadowRay and Langflow.
Long-lived provider keys in code and clients: keys in front-end bundles, mobile apps or public repositories, with no spend cap, leading to LLMjacking and large bills. The Sourcegraph token leak turned a leaked admin token into a free LLM proxy.
Testing once: safety evaluated on one model version; then the model, prompt or fine-tune changes and nobody re-runs the tests.

Practitioner checklist

Maintain an inventory of LLM applications, models, versions, providers, keys, data sources and tools.
Remove credentials, role structures and business limits from system prompts and enforce them in code.
Filter retrieved data by the caller's permissions before it enters the context window.
Encode all output for its destination, set a strict CSP, block remote image rendering and use parameterised queries.
Run any model-generated code in a sandbox with no secrets and no outbound network.
Set per-identity token limits, timeouts, quotas and spend alerts, and restrict logprobs and logit_bias.
Require authentication on every inference, gateway and orchestration endpoint, and keep provider keys in a secrets manager.
Run input and output guardrails, tuned in monitor mode before enforcement, plus abuse monitoring at conversation level.
Red team jailbreak, extraction and output injection paths before release and after every model, prompt or fine-tune change.
Verify model and adapter provenance with signatures and hashes, and record them in an AI-BOM.

LLM Security Guide: Prompt Injection, Jailbreaks and Data Leakage