Prompt injection attacks in LLMs: are your controls keeping up?

TL;DR: Prompt injection can override an LLM’s original instructions, causing incorrect outputs, data leakage, unauthorized access, and unintended actions, according to Fiddler’s analysis of red-teaming patterns and controls. The real issue is not just model brittleness, but the governance gap between input handling, output trust, and operational monitoring.

NHIMG editorial — based on content published by Fiddler: Evaluate LLMs Against Prompt Injection Attacks Using Fiddler Auditor

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.

Questions worth separating out

Q: What breaks when prompt injection defenses are not in place for LLMs?

A: When prompt injection defenses are weak, attacker-controlled text can override the model's intended instructions, especially when the text arrives through retrieved content or external documents.

Q: Why do LLMs make access and data governance harder for security teams?

A: LLMs complicate governance because they can combine user input, retrieved data, and tool output into one decision path.

Q: How do teams know whether prompt injection controls are actually working?

A: Look for end-to-end visibility across prompts, retrieved content, memory, tool calls, and outputs, plus evidence that blocked actions stay blocked under realistic test cases.

Practitioner guidance

• Define trust boundaries for every prompt path Separate system instructions, user input, retrieved context, and tool output so untrusted text cannot override policy.
• Test AI workflows with adversarial perturbations Red-team prompts before production using paraphrase, encoding, and context poisoning scenarios that reflect real attack paths.
• Limit tool authority for LLM-connected systems Grant the minimum function scope needed for the workflow and require human approval for high-impact actions such as record changes, credential retrieval, or external communication.

What's in the full article

Fiddler's full blog covers the operational detail this post intentionally leaves for the source:

• The exact custom transformation pattern used to generate adversarial prompt variants and test model robustness.
• The code structure for implementing the prompt injection transform in the Auditor framework.
• A concrete walkthrough of how the model responded across five injected prompt variations.
• The red-teaming workflow used to evaluate OpenAI text-davinci-003 against manipulated instructions.

👉 Read Fiddler's deep dive on evaluating LLMs against prompt injection attacks →

Prompt injection attacks in LLMs: are your controls keeping up?

Explore further

View Full Forum →  |  NHI Foundation Course →