How should security teams implement guardrails for generative AI apps that use RAG and agentic frameworks?

Start by placing controls at the prompt and response boundaries, where sensitive data and malicious instructions enter or leave the model. A practical programme combines input inspection, output filtering, redaction, and policy enforcement so developers can keep working without exposing secrets or regulated data. The goal is to reduce leakage and jailbreak risk without turning AI adoption into a blocked workflow.

Guardrails at the RAG and Agent Boundaries

For generative AI apps that combine retrieval and agent execution, the important control point is not the model alone but the full path from retrieved content, tool calls, and user input to the final response. Security teams should treat RAG as a data-exposure problem and agentic frameworks as an authority-expansion problem, because each can turn ordinary prompts into a route for leakage, prompt injection, or unsafe action.

That is why the first guardrails should inspect what enters the model, constrain what it is allowed to retrieve or call, and verify what leaves the system before it reaches users or downstream tools. The right design reduces the chance that hidden instructions in documents, or excessive tool scope in an agent, can override policy. In practice, many security teams discover the weakest control after a retrieval source or tool integration has already been trusted by default.

One useful reference point is the OWASP Agentic AI Top 10, which is especially relevant where the app can take actions rather than only generate text.

How Controls Work Across Retrieval, Prompts, and Actions

Effective guardrails work as a sequence of checks rather than a single filter. At the retrieval layer, the app should limit which documents, chunks, indexes, or connectors can be surfaced to the model, and it should reject content that looks like injected instructions, unexpected schemas, or high-risk data classes. At the prompt layer, the system should distinguish user intent from retrieved text so that the model is less likely to treat untrusted content as instructions. At the response layer, the app should remove secrets, personal data, and policy-sensitive content before display or export.

For agentic frameworks, the control question changes from “what may the model say?” to “what may the agent do?” That means tool use needs allowlisting, scope limits, approval gates, and logging that make every action attributable. If an agent can search, write, send, or modify data, then the safe design is to give it the minimum capability needed for the task and to separate read-only actions from state-changing ones.

Constrain retrieval so only approved sources and document classes can reach the model.
Sanitise prompts and retrieved text so untrusted instructions are treated as data, not policy.
Filter outputs for leakage, policy violations, and unsafe operational instructions.
Require human approval or tighter checks before any agent performs external actions.
Log retrievals, tool calls, and policy decisions so investigations can reconstruct model behaviour.

The NIST AI Risk Management Framework is helpful here because it frames these checks as governance and measurement problems, not just engineering tasks. Where the workflow depends heavily on tool execution, the MITRE ATLAS adversarial AI threat matrix helps teams think through how prompt injection, manipulation, and abuse may unfold across the lifecycle.

These controls fail when teams rely on a single “safe prompt” layer while leaving retrieval sources, plugins, or tool permissions broadly trusted.

Where Guardrails Need Tuning, Not Just Tightening

Tighter AI guardrails often improve containment, but they also increase friction for developers and can suppress legitimate retrieval or action flows, so organisations have to balance safety against usability. The tradeoff is most visible in agentic systems, where a control that is too strict can make the agent unusable, while a control that is too loose can make every downstream action a security concern.

RAG-heavy applications also vary in how much trust they can place in source content. Internal knowledge bases, regulated records, and user-uploaded files should not all be handled the same way, because the risk is not identical. The guidance is to apply stronger filters to uncurated or externally influenced content, while allowing lower-friction retrieval where the corpus is already governed and well-owned. There is still industry disagreement on how much prompt-injection content can be reliably detected before retrieval, so teams should treat pre-retrieval scanning as useful but not complete.

For organisations standardising their AI programme, the NIST AI 600-1 Generative AI Profile is a practical way to anchor guardrails to generative-AI-specific risks, while the NIST Cybersecurity Framework 2.0 remains useful for mapping the surrounding governance, resilience, and monitoring functions.

The approach breaks down when teams try to apply the same policy threshold to every retrieval source and every agent action, because that ignores trust differences and leads to either unsafe exposure or unusable systems.

Risk and Threat Considerations

RAG and agentic frameworks create a combined exposure path: untrusted content can influence model behaviour, and excessive tool authority can turn that influence into real-world action. The material risk is prompt injection, data leakage, and unsafe automation, especially where retrieved content, user input, and tool outputs are all allowed to interact without clear trust boundaries.

Failure mechanism: A malicious or contaminated retrieval source can place instructions into the model context, causing the system to override policy, reveal sensitive data, or invoke tools in ways the operator did not intend. The same weakness can appear when an agent is given broad connector access or write privileges without approval gates, because the model may faithfully execute harmful or unexpected actions under the guise of normal task completion.

Impact: Organisations can expose confidential information, create unauthorised changes in downstream systems, send unapproved messages, or lose confidence in the integrity of AI-assisted workflows. In higher-trust environments, the problem can become a governance failure as well as a security one, because teams may not be able to prove which content influenced which action.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and CIS Controls v8 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Agentic AI Top 10	A1 — Prompt Injection	RAG and agentic apps are directly exposed to injected instructions.
	A2 — Data Exfiltration	Guardrails must prevent leakage from prompts, outputs, and tools.
	A3 — Excessive Agency	Agentic frameworks need constrained tool scope and approvals.
Recommendation — Treat retrieved content as untrusted and isolate it from system instructions. Filter prompts and outputs to block sensitive-data disclosure paths. Restrict tool permissions and require approval before state-changing actions.
NIST AI RMF	GOV — Govern	Guardrails require AI governance, accountability, and policy ownership.
	MAP — Map	Teams must understand where retrieval, prompts, and actions create risk.
	MEASURE — Measure	Guardrail effectiveness depends on observable policy and leakage signals.
Recommendation — Assign accountable owners and define policy for model, retrieval, and tool use. Inventory data sources, tools, and trust boundaries before enabling the app. Measure leakage, override rates, and unsafe-action attempts to validate controls.
MITRE ATLAS	AML.T0011 — Prompt Injection	Prompt injection is a core adversarial pattern in RAG and agent workflows.
	AML.T0057 — Model Denial of Service	Guardrail abuse can degrade availability and disrupt AI workflows.
Recommendation — Hunt for prompt-injection patterns in retrieved content and agent sessions. Monitor for abuse patterns that exhaust context, tokens, or tool capacity.
CIS Controls v8	6 — Access Control Management	Agent tools and retrieval connectors need least-privilege access governance.
	8 — Audit Log Management	Session-level logging is essential for reconstructing AI decisions and actions.
Recommendation — Limit connector and tool access to only the privileges the workflow requires. Log retrievals, prompt decisions, and tool calls for investigation and review.

Practitioner Guidance

What to prioritise: Start with the highest-risk junctions, not the broadest policy. For most teams that means untrusted retrieval sources, tool connectors that can change state, and any path that can surface regulated or confidential data.

What to verify: Confirm that the system can distinguish trusted instructions from retrieved text, and that every external action has an explicit permission model. If you cannot trace why a tool was called, the guardrails are not yet strong enough for production use.

What good looks like: A secure deployment keeps read paths and action paths visibly separate, with logging that shows what was retrieved, what was filtered, and what the agent was allowed to do. The practical test is whether security teams can review a session and reconstruct the decision chain without guessing.

Practitioner takeaway: The safest AI guardrail programme is usually one that narrows authority more than it tries to “detect bad prompts,” because access scope and trust boundaries are easier to govern than model behaviour alone.