Test the full path, not just the chat box. Include direct prompt injection, indirect injection through documents and webpages, memory abuse, encoded text, and image-based prompts. The goal is to see whether the model can be pushed into unsafe tool use or disclosure when inputs look trusted but are not.
Why This Matters for Security Teams
Prompt injection is not just a chat abuse problem. It is a control-bypass problem that can turn a harmless-looking input into unsafe tool use, disclosure of sensitive context, or unauthorised action by an AI system. Security teams that only test the visible prompt surface miss the more important question: whether the model, the retrieval layer, and any connected tools will still behave safely when the input is adversarial.
This matters because GenAI systems increasingly operate across documents, emails, web pages, ticketing systems, and internal knowledge stores. A malicious instruction can arrive indirectly, be buried inside trusted content, or be encoded in ways that defeat superficial filters. Current guidance suggests evaluating the entire execution path, including model routing, retrieval augmentation, memory, plugins, and output handling. The OWASP Agentic AI Top 10 is useful here because it treats unsafe autonomy, tool abuse, and instruction conflicts as first-class risks rather than edge cases.
In practice, many security teams discover prompt injection only after a connected agent has already exposed data or taken an unwanted action, rather than through intentional test design.
How It Works in Practice
Effective testing starts by building scenarios that resemble real attack paths, not just adversarial one-liners in a chat window. Testers should validate direct prompt injection, indirect injection through retrieved content, memory poisoning, encoded or obfuscated instructions, and multimodal prompts hidden in images or screenshots. The objective is to observe whether the system can be manipulated into ignoring system instructions, escalating privilege through tools, or leaking hidden context.
A useful test plan usually covers four layers:
- Input handling: can the system recognise hostile instructions in user messages, attachments, and retrieved text?
- Instruction hierarchy: does it preserve system and developer intent when lower-trust content conflicts with policy?
- Tool governance: does the model ask for confirmation before risky actions, or can it call tools freely?
- Output validation: does the application block sensitive disclosures, unsafe code, or policy-violating actions before they leave the boundary?
For agentic or tool-using systems, test sequences should chain prompts with realistic tasks, such as summarising a document that contains hidden instructions, then asking the agent to act on the summary. That exposes whether the model obeys the hidden instruction at inference time. The NIST AI 600-1 GenAI Profile is helpful as a governance reference because it emphasises mapping controls to model risk, use context, and downstream impact.
Testing should also include logging and detection. Security teams need visibility into prompt content, retrieval sources, tool calls, and refusals so they can distinguish a successful attack from a benign model failure. Where the system uses retrieval-augmented generation, the trust boundary must include the indexed corpus, not just the live prompt. These controls tend to break down when the model can invoke external tools without explicit policy checks because the attack then shifts from text manipulation to unsafe action execution.
Common Variations and Edge Cases
Tighter prompt filtering often increases false positives and operational friction, requiring organisations to balance safety against usability and support burden. There is no universal standard for prompt injection testing yet, so best practice is evolving and should be adapted to the system’s autonomy, data sensitivity, and tool access.
Some environments need additional emphasis. In enterprise search and RAG systems, indirect injection is often more important than direct jailbreaking because the adversary can hide instructions inside content that already appears trusted. In multimodal systems, image-based prompt injection and OCR-transcribed text create another route around text-only filters. In long-running agents, memory abuse becomes a persistence risk because a single malicious instruction may influence later actions long after the original input was forgotten.
For high-risk deployments, current guidance suggests pairing red-team testing with control validation: least-privilege tool scopes, explicit human approval for sensitive actions, provenance checks on retrieved content, and post-processing that strips or quarantines untrusted instructions. The right test is not whether the model can be tricked once, but whether the surrounding system prevents a trick from becoming an incident. If the system accepts external content, keeps memory across sessions, and can act through tools, prompt injection resistance should be treated as a continuous assurance problem rather than a one-time test.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 | Prompt injection is a core agentic AI failure mode involving tool abuse and instruction hijacking. |
| NIST AI RMF | AI RMF frames prompt injection as a lifecycle risk across governance, mapping, measurement, and management. | |
| NIST AI 600-1 | The GenAI profile helps translate model risk into practical controls for unsafe outputs and misuse. | |
| NIST CSF 2.0 | PR.DS, DE.CM | Prompt injection testing needs data protection and continuous monitoring controls around the AI stack. |
| MITRE ATLAS | AML.TA0001 | ATLAS covers adversarial AI tactics including inputs designed to subvert model behaviour. |
Assess prompt injection risk across the AI lifecycle and tie findings to accountable risk treatment.
