How should security teams test LLM applications that include RAG pipelines and agents before production deployment?

Security teams should test the full LLM system, not just the model endpoint. That means exercising prompts, retrieval paths, tool calls, and API integrations with context-aware attacks that reflect the application’s real inputs and guardrails. The goal is to surface application-specific failure modes such as context injection, document leakage, memory poisoning, and unsafe tool escalation before users or attackers encounter them.

Testing the whole LLM system, not just the model

Pre-production testing for LLM applications needs to reflect the actual product architecture, because the main failure modes often sit outside the base model. A RAG pipeline can retrieve the wrong context, expose sensitive source material, or elevate untrusted text into the prompt. An agent can also take unsafe actions if tool permissions, routing logic, or guardrails are weak. For that reason, teams should test prompts, retrieval, ranking, context assembly, tool use, and integration points as one system rather than as isolated components.

That system view matters because the threat surface changes once the application can fetch data, hold state, or act on behalf of a user. A model that looks safe in a sandbox can still fail when it receives poisoned documents, conflicting instructions, or ambiguous tool results. The practical benchmark is whether the application behaves safely under realistic inputs, not whether the model answers well on canned prompts. The OWASP Agentic AI Top 10 is a useful external reference point for the failure classes that emerge once tools, memory, and external actions are part of the design.

In practice, many security teams only discover the weak link after retrieval, memory, or tool access has already been wired into production workflows.

How to exercise RAG paths, tools, and guardrails before release

Effective testing should cover the paths where the application can be manipulated, not just the paths it is expected to follow. For RAG, that means checking whether the system retrieves authoritative sources, whether low-trust text can override higher-trust instructions, and whether the prompt builder leaks material that should remain hidden from the user. For agents, it means validating whether tool calls are constrained by explicit policy, whether the agent can be coerced into over-broad actions, and whether the output layer blocks unsafe escalation.

A useful test plan usually combines adversarial prompts with workflow-aware probes. The adversarial prompts should reflect what the application actually does: requests that try to inject instructions into retrieved documents, prompts that ask for hidden context, and prompts that try to steer an agent toward privileged or destructive actions. The workflow-aware probes should verify the intended sequence of operations: what gets retrieved, what is retained in memory, what is passed to tools, and what is returned to the user. When those stages are tested separately, teams miss the compound failures that appear only when retrieval, memory, and action are chained together.

Security teams should also test the failure boundaries, not only the success path. That includes malformed input, ambiguous retrieval results, empty or irrelevant context, and tool responses that conflict with the model’s plan. Strong systems degrade safely when context quality is poor; weak systems improvise. The NIST AI Risk Management Framework is relevant here because it encourages teams to evaluate AI systems through govern, map, measure, and manage activities rather than treating the model as the only object of testing.

Verify that retrieved passages cannot silently override higher-trust instructions.
Check that tool permissions are narrowed to the minimum action set required for the use case.
Test whether memory or session state can persist poisoned instructions across turns.
Confirm that unsafe requests are blocked before the tool layer, not only after the model generates a risky plan.

Where teams usually go wrong is assuming that a benign benchmark or a clean demo transcript proves the pipeline is safe. It does not, because real failure appears when multiple weak controls interact under adversarial input.

Where RAG and agent testing gets messy in production-like edge cases

Tighter testing often increases operational overhead, because realistic evaluation requires representative data, representative tools, and representative failure conditions. That creates a trade-off between speed and confidence, especially when the application depends on rapidly changing documents or external systems. The most common edge cases are not exotic attacks; they are trust-boundary mistakes, such as a low-quality document being treated as authoritative, a retrieval hit being over-valued because it is recent, or an agent being given a tool that is broader than the task actually requires.

There is also a governance question about how much of the production environment should be mirrored in test. Teams do not need live secrets or unrestricted integrations to test safely, but they do need enough realism to expose unsafe orchestration. If the test environment omits the very permissions, data classes, or retrieval sources that matter, the result is only a model demo. Where the question is specifically about agentic workflows, the OWASP Top 10 for Agentic Applications 2026 and MITRE ATLAS both help distinguish ordinary prompt quality problems from adversarial manipulation of the agent’s action chain. Anthropic’s report on AI-orchestrated cyber espionage is useful as a reminder that tool-enabled systems can be abused for coordinated abuse when controls are too permissive.

The boundary case that breaks this guidance is a static chatbot with no retrieval, memory, or external actions, because then the right test focus shifts back toward prompt behaviour and output safety rather than pipeline abuse.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
NIST AI RMF	MEASURE — Measure	RAG and agent tests should evaluate system behavior, not only model output.
Recommendation — Measure retrieval, tool-use, and guardrail failures under adversarial prompts before release.
OWASP Agentic AI Top 10	A1 — Input and Context Injection	Prompted retrieval and agent workflows are exposed to injected instructions and context abuse.
	A2 — Unsafe Tool Use	Agents can be pushed into harmful or over-broad tool actions if permissions are weak.
	A4 — Data Leakage	RAG pipelines can surface source material or hidden context to users or attackers.
Recommendation — Test for injected instructions in prompts, retrieved content, and agent context paths. Constrain and exercise tool calls to confirm the agent cannot trigger unsafe actions. Probe for unintended disclosure across retrieval, memory, and response generation.
MITRE ATLAS	AML.TA0001 — Reconnaissance	Adversarial testing should reflect how attackers probe AI systems for weak points.
Recommendation — Map probes and abuse cases to adversarial AI techniques and expand coverage iteratively.

Practitioner Guidance

What to prioritise: Test the joins between retrieval, memory, and tool execution first, because that is where the highest-impact failures usually appear. A clean model response is not evidence of a safe application if the surrounding orchestration can still be steered.

What to verify: Confirm that each control boundary enforces its own trust decision. Retrieval should not grant authority, memory should not preserve hostile instructions, and tools should not execute actions that the user or workflow would not otherwise be allowed to request.

Decision rule: If a test only proves the model can answer well in isolation, treat it as incomplete. If it shows the system resists manipulation after retrieval and before action, treat it as meaningful pre-production evidence.

Practitioner takeaway: The safest deployment decision comes from testing the orchestration layer as aggressively as the model, because that is where RAG and agents turn from content systems into security-relevant systems.