AI prompt injection defenses still fail when models reveal secrets

At a glance

What this is: This is Lakera’s walkthrough of the Gandalf prompt-injection challenge, showing how LLM defenses can improve across levels while still failing to prevent secret leakage.

Why it matters: It matters because IAM and security teams increasingly need to govern what AI systems can be induced to reveal or do, not just what they are told not to disclose.

Context

Prompt injection is a control problem, not just a content problem. When an LLM can be persuaded to reveal a secret or echo sensitive instructions, the security question is no longer whether the model received the right system prompt, but whether runtime controls can withstand adversarial conversation.

Lakera’s Gandalf example frames that gap clearly through a password-reveal challenge with seven levels of increasing defence. The article is most relevant to AI agent and LLM governance because it shows how input filtering, output filtering, and transcript review can still fail when the model can be induced to cooperate with the attacker.

The underlying lesson extends to any AI workflow that handles sensitive information, from customer data to account balances and patient records. A model that can be manipulated into disclosure behaves like a governed identity surface, even when it is not a human or a classic NHI.

Key questions

Q: What breaks when LLM prompt filters rely on keywords alone?

A: Keyword-only filters fail when an attacker asks indirectly, translates the request, or splits the secret into fragments the model can still reconstruct. The guard sees safe text, but the model still understands the disclosure intent. That is why prompt injection often succeeds even when obvious secret terms are blocked.

Q: Why do transcript-based prompt injection checks still miss leaks?

A: Transcript checks can miss leakage when the model reveals a secret through paraphrase, partial disclosure, encoding, or reconstruction across several turns. The weakness is not just string matching, but assuming the exact secret must appear verbatim for compromise to occur. Adversaries rarely need that.

Q: How do you know if an LLM guardrail is actually working?

A: A guardrail is working when it catches harmful content consistently without blocking a large share of legitimate requests. Look for stable precision and recall on realistic samples, low latency at production volume, and category-level performance that matches your threat model. If safety outcomes vary sharply by task, the control is not reliable enough for broad deployment.

Q: What should security teams do when employees are using public LLMs with sensitive data?

A: They should assume the data is at risk and enforce controls immediately. That includes restricting sensitive inputs where possible, monitoring prompts and outputs, and training users on acceptable use. Public LLM usage turns ordinary workflows into disclosure paths, so the response has to combine policy, technical controls, and user awareness.

Technical breakdown

Why keyword blocking fails against prompt injection

Keyword blocking only catches the most obvious form of elicitation, such as asking for a password by name. Gandalf shows that attackers can ask indirectly, translate the request, split it into halves, or reframe it so the model still interprets the intent while the guard does not. That makes keyword checks a shallow content filter rather than a true control on model behaviour. In practice, prompt injection succeeds because the attacker is not just sending text, but shaping the model’s interpretation of that text across multiple turns.

Practical implication: treat keyword blocking as a narrow filter, not as a sufficient control for sensitive model outputs.

How transcript-based leakage checks still miss model-mediated disclosure

Transcript checking is stronger because it inspects the conversation instead of a single prompt. Even so, Gandalf shows that transcript-aware defences can still miss encoded answers, partial disclosures, and outputs that reveal the secret without spelling it out directly. The core failure is that language models can satisfy the user’s intent while avoiding the exact string the guard is looking for. That turns the defence into a detection problem with high false negatives whenever the secret can be paraphrased, obfuscated, or reconstructed.

Practical implication: evaluate controls against indirect disclosure, not just exact-string matching.

Why LLM guardrails behave like runtime policy, not static prevention

The Gandalf levels illustrate a broader architectural point: once an LLM is allowed to receive arbitrary user input, its behaviour is probabilistic and adversarially steerable. The model does not simply obey a fixed access rule in the way a traditional application would. Instead, every turn becomes a decision surface where input, context, and output all interact. That makes prompt injection a runtime governance issue, closer to authorisation enforcement than to content moderation. The relevant design question is whether the system can stop unsafe action after interpretation has already begun.

Practical implication: design controls around runtime decision points, not only around pre-approved prompts.

Threat narrative

Attacker objective: The attacker’s objective is to induce the model to reveal a protected secret and prove that the guardrail stack cannot reliably contain disclosure.

1. Entry occurs when the attacker submits crafted prompts that steer the model toward a secret without asking for it in plain language.
2. Credential-like exposure follows when the model discloses the password, a partial password, or an encoded version that can be reconstructed.
3. Impact occurs when the attacker can reliably extract protected information from the conversation despite layered guards.

Breaches seen in the wild

• iOS apps leaking hard-coded secrets: Cybernews found 71% of 156,080 iOS apps leak hard-coded secrets, with open cloud storage and Firebase databases exposing user data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection is an authorisation failure disguised as a content problem: The article shows that the real issue is not whether the model can recognise bad words, but whether it can be manipulated into taking an unsafe disclosure path. Traditional content filters treat the prompt as text to classify; adversarial prompting turns the prompt into a control channel. The practitioner conclusion is that LLM governance must focus on runtime decision boundaries, not just safe-input lists.

Blocking the secret token does not block the secret: Gandalf demonstrates that partial answers, paraphrases, transliterations, and encoded references can still carry the protected value. That means the control assumption is too narrow if it only watches for exact leakage. For identity and access teams, the lesson is familiar: if disclosure can be re-expressed, the policy is already being bypassed.

Model safety rules fail when attackers control the conversation state: Each level shows the attacker shaping context over multiple turns until the model cooperates. This is why LLM guardrails behave more like session policy than static filtering. The practitioner implication is that governance must account for context accumulation, not just one-shot request screening.

Runtime model governance is now part of the identity perimeter: When an LLM can be induced to reveal protected information, the model itself becomes a governed decision surface. That brings the problem into the same category as access enforcement, disclosure control, and delegated authority, even though the actor is not human. Practitioners should treat AI output control as part of identity and access architecture, not as a separate niche concern.

Named concept, transcript leakage boundary: The article illustrates a transcript leakage boundary, where the secret is not lost through a single prompt but through the cumulative conversation that a guard may fail to interpret correctly. That boundary matters because it defines where static text checks stop working and runtime supervision must begin. Teams should assume that conversation history is part of the attack surface, not just a record.

What this signals

Transcript leakage boundary: The important control line is not the prompt alone, but the full exchange that shapes the model’s next response. When attackers can steer the conversation over multiple turns, disclosure controls need to supervise context accumulation as carefully as they supervise final output.

Prompt injection also shows why AI security belongs alongside identity governance. The model is not a person, but it can still be induced to act outside its intended disclosure boundaries, which means teams have to manage runtime authority, not just static prompt hygiene.

For practitioners

• Map disclosure paths by conversation stage Identify where the model can be steered from benign context to secret exposure, then review each stage for gaps in prompt filtering, output review, and transcript handling.
• Test for indirect elicitation patterns Red-team prompts that request the secret through translation, acrostics, partial reconstruction, roleplay, and other paraphrase paths instead of direct asks.
• Treat transcripts as governed security artefacts Apply review and retention rules to prompt history because the attack often lives in the conversation sequence, not in one isolated message.
• Separate disclosure control from keyword moderation Measure whether the model can still reveal protected data when the exact secret term never appears in the user input or output.

Key takeaways

• Prompt injection remains a live control problem because LLMs can be steered into disclosure even when direct requests are blocked.
• Lakera’s Gandalf walkthrough shows that indirect elicitation, encoding, and transcript-aware attacks can bypass layered defences.
• Security teams need runtime disclosure controls that evaluate conversation context, not just keyword presence.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Transcript Leakage: Transcript leakage is the exposure of protected information through the full conversation history rather than through a single obvious prompt or response. It matters because a model can reveal or reconstruct secrets across multiple turns, making exact-string filters insufficient.
• Runtime Disclosure Control: Runtime disclosure control is the set of checks that govern what an AI system may reveal while it is actively processing a conversation. In practice, it has to evaluate context, intent, and output behaviour, not just the presence of banned words or phrases.
• Conversation State: Conversation state is the persistent or semi-persistent record an AI system uses to continue a dialogue across turns. It includes message history, session metadata, and any cached reconstruction data. If that state is incomplete or misbound, the system may generate answers from the wrong context.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.