System prompt leakage exposes the hidden trust gap in LLM apps

At a glance

What this is: StackHawk explains that system prompt leakage exposes sensitive instructions, credentials, and business logic when attackers get LLMs to reveal the prompts themselves.

Why it matters: This matters because security teams cannot treat prompts as a secrecy layer, and any credentials or access rules embedded there become part of the attack surface for AI applications.

Context

System prompt leakage is a governance failure, not just a model-output bug. It happens when teams use LLM instructions to store secrets, access rules, or internal logic that should live in deterministic application controls. In practice, the model becomes a disclosure path rather than a security boundary, which creates a direct intersection with IAM, NHI credential handling, and authorization design.

In AI applications, the prompt is often treated as configuration, but the article shows why that boundary is unsafe when the configuration contains credentials or role logic. If the prompt can be elicited or inferred, the hidden trust model becomes visible to an attacker. That pattern is typical wherever developers blur instruction-setting with control enforcement.

Key questions

Q: What breaks when sensitive information is stored in system prompts?

A: When sensitive data sits inside system prompts, the model becomes a disclosure path instead of a control boundary. Attackers can extract credentials, internal endpoints, or access rules and then use that information to target backend systems or bypass security checks. The failure is architectural because the prompt was never a safe place for secrets.

Q: Why does prompt leakage create a broader security risk than simple disclosure?

A: Prompt leakage is risky because the leaked content often includes the operating rules of the application, not just text. Once an attacker sees thresholds, role logic, or integration details, they can shape abuse to fit the system’s own assumptions. That turns information disclosure into a practical privilege and reconnaissance problem.

Q: How can security teams tell whether their LLM app is relying on prompt secrecy?

A: Look for prompts that contain secrets, internal service names, access policy text, or moderation instructions that the application still depends on at runtime. If removing those values would break the security model, the app is relying on prompt secrecy. Good controls survive prompt exposure because they are enforced elsewhere.

Q: How should teams redesign LLM apps so prompt leakage is less dangerous?

A: Keep prompts focused on behaviour and move sensitive material into deterministic services that handle identity, authorization, and secret retrieval. Use application logic for access control, rotate secrets independently, and assume the prompt may be observed by an attacker. That design reduces the value of any leaked instruction set.

Technical breakdown

Why system prompts are not a secure control plane

A system prompt is an instruction set, not a protected vault. If developers embed API keys, internal endpoints, access rules, or moderation logic inside it, the model may reveal those details under prompt injection or direct questioning. The problem is not limited to exact prompt disclosure. Even partial leakage can reveal enough structure for recon to target backend services, infer privilege boundaries, or bypass filters. In security terms, the prompt becomes a soft control that cannot guarantee confidentiality or authorization.

Practical implication: keep secrets and enforcement logic outside the prompt and treat prompt content as potentially observable.

How prompt injection turns disclosure into privilege risk

Prompt injection works because the model follows user-supplied instructions or manipulative phrasing in ways developers did not intend. When the system prompt contains credentials or role definitions, injection can convert a disclosure into a broader access problem. An attacker who learns internal service names, token formats, or override rules may not need the prompt again. They can use that information against backend APIs, workflow gates, or human review processes. The real architectural failure is assuming the model can safely both describe and enforce access policy.

Practical implication: enforce authorization in application logic and backend services, not in prompt language.

Why LLM apps expose an identity and secrets governance gap

The article’s most important insight is that prompt leakage is really a secrets and identity design problem. API keys, bearer tokens, role descriptors, and transaction override logic should not depend on obscurity inside model instructions. Once those values sit in the prompt, they are no longer governed as credentials with lifecycle controls such as inventory, rotation, revocation, and scoped use. For NHIs, that creates an unhealthy overlap between machine identity management and AI instruction design, which is exactly where control failures tend to hide.

Practical implication: inventory every secret and access rule used by the application, then remove anything sensitive from model-visible instructions.

Threat narrative

Attacker objective: The attacker wants to extract hidden instructions and use the disclosed credentials, rules, and system details to undermine access control and target backend services.

1. Entry occurs when an attacker prompts the model to repeat or reveal its exact instructions, often using prompt injection or configuration-style questions.
2. Credential access follows when the leaked system prompt contains API keys, bearer tokens, connection strings, or override codes that were never meant to be user-visible.
3. Impact comes when the attacker uses the disclosed details to target backend services, bypass security filters, or structure abuse around known business rules and access thresholds.

Breaches seen in the wild

• Dropbox Sign breach 2024: A compromised back-end service account gave attackers Dropbox Sign customer data, including API keys, OAuth tokens and MFA information.
• Hugging Face Spaces breach 2024: Unauthorised access to Hugging Face Spaces may have exposed secrets users stored for AI apps; tokens were revoked and org tokens removed.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

System prompt leakage is a secrets management failure disguised as an AI content issue. When teams place credentials, override codes, and internal endpoints inside prompts, they have already lost the boundary that secrets management is supposed to enforce. That turns the LLM into a disclosure channel for machine credentials and internal control logic, which is structurally incompatible with good NHI governance. Practitioners should treat prompt content as non-secret by default.

The hidden trust gap in LLM apps is that developers expect prompts to behave like policy, storage, and enforcement at the same time. Those roles should never be combined. Once a prompt contains both instructions and sensitive data, any leakage becomes a governance event, not just an application defect. The practical conclusion is simple: the model may guide behaviour, but deterministic systems must own authorization and credential use.

System prompt leakage creates a predictable blast-radius problem because disclosure and abuse happen in the same trust domain. The attacker does not need full compromise to gain value from leaked architecture, access thresholds, or moderation criteria. That makes the control failure broader than classic sensitive-information disclosure, because the exposed material also reveals how to evade safeguards. Security teams should assume prompt exposure can accelerate both reconnaissance and privilege abuse.

Named concept: prompt secrecy debt. This is the accumulation of hidden business logic, access rules, and credentials inside AI instructions that the organisation can no longer defend operationally. It is a form of governance debt because the longer those values stay in prompts, the harder they are to inventory, rotate, or revoke in a controlled way. Practitioners should eliminate prompt secrecy debt before it becomes normalised across production systems.

From our research library:

• 40 percent of financial and software companies have already deployed agentic AI systems, and deployments are expected to double by 2028.
• Read next: API Key Management Guide

What this signals

Prompt secrecy debt: organisations that let prompts accumulate credentials, thresholds, and access rules are building a control problem that becomes harder to unwind with every release. The better model is to treat the prompt as behaviour guidance only, while secret retrieval and authorization remain external and auditable.

Security teams should expect AI applications to be tested for disclosure paths the same way they already test for injection and broken authorization. When prompts expose backend logic, the issue is not merely model misbehaviour, but weak separation between AI instructions and the controls that should govern identity and secrets.

The practical shift is to review LLM applications through a governance lens: what lives in the prompt, what lives in code, and what lives in the secret manager. That distinction matters because prompt exposure should never also mean credential exposure.

For practitioners

• Separate prompts from credentials Move API keys, tokens, connection strings, and override codes out of system prompts and into controlled secret stores or runtime configuration that the model cannot directly expose.
• Enforce authorization outside the model Implement role checks, transaction approval rules, and security filters in application code and backend services rather than in prompt instructions.
• Review prompts for hidden business logic Audit production prompts for internal thresholds, moderation rules, and access descriptions that would help an attacker infer system behaviour or target sensitive workflows.
• Test prompts with adversarial inputs Include prompt-injection and disclosure cases in security testing so teams can see whether the model reveals instructions, credentials, or internal architecture details under pressure.

Key takeaways

• System prompt leakage is dangerous because it often exposes both confidential instructions and the credentials or access rules embedded inside them.
• The main failure is architectural: prompts are not a secure place to store secrets or to enforce authorization decisions.
• Teams should externalize secrets, enforce access in deterministic services, and test LLM applications for prompt-injection disclosure paths.

Key terms

• System Prompt Leakage: System prompt leakage is the exposure of hidden prompt content to users or attackers. The real security problem is usually not the prompt itself, but the secrets, policy logic, and internal architecture details placed inside it. If those details are sensitive, they should live in code or secrets management instead.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Prompt Secrecy Debt: Prompt secrecy debt is the accumulation of sensitive configuration, business rules, and credentials inside AI instructions that were never meant to serve as a control boundary. It creates operational risk because the longer those values remain embedded in prompts, the harder they are to inventory, govern, and revoke.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity control to the broader security architecture their programmes depend on.