AI Guardrails and AI Firewalls Guide

Guardrails and AI firewalls are sold as the control that makes LLM applications safe, but most of them are probabilistic classifiers that reduce attack volume rather than enforce a boundary. This guide covers the controls themselves: input and output filtering, where enforcement can sit (model, application SDK or gateway), detection techniques, PII redaction, guardrails on agent tool calls, their limits (bypass, false positives, latency), and how to roll out, test and measure them. It is written for AI platform teams, application security engineers and architects choosing or tuning guardrails. Attack mechanics are in the Prompt Injection Guide and the LLM Security Guide; this page contains no attack strings.

Key takeaways

• A guardrail is a detective control unless code enforces it. Classifiers lower attack success; permissions, schemas and egress rules enforced outside the model hold when a classifier misses.
• Placement decides what a guardrail can see. A gateway sees every call but not entitlements or tool effects; an in-app SDK sees context but only where adopted; model-native filters see neither your policy nor your data.
• Screen tool calls, not only chat. For agents, the guardrail that matters most checks each proposed action against the agent identity, the delegated user and the target before it runs.
• Every guardrail has an error budget. Measure misses with red team suites and false positives with real traffic, per category and language, and set thresholds from data, not vendor defaults.
• Start in detect mode, then block. Shadow mode exposes false positives and latency before users feel them; a per-policy kill switch gives rollback without a redeploy.

What guardrails and AI firewalls are

An OpenAI cookbook article describes a guardrail as "a generic term for detective controls that aim to steer your application". "AI firewall" is a market term, not a standard one: usually a guardrail engine deployed as a proxy or gateway in front of one or more models, with rate limiting, logging and data loss prevention added. The useful distinction is where a control runs and whether its decision is deterministic or probabilistic.

NVIDIA's open-source NeMo Guardrails defines five stages (input, retrieval, dialog, execution and output rails) that work as a taxonomy for any product; execution rails, for example, "control and validate tool/function calls, their arguments, and results".

Where to enforce: model, SDK or gateway

Most organisations end up with several enforcement points, so decide which layer owns which policy rather than running the same classifier three times on the latency path.

• Gateway for breadth, SDK for depth: identity, quotas, logging and coarse content policy in the gateway; retrieval, grounding and tool policy where application context exists.
• Treat the gateway as privileged: it holds provider keys and sensitive prompts. See the LLMjacking Guide.
• Check what happens to a verdict: the standalone Azure Prompt Shields API "reports attackDetected" and leaves blocking to the application; in Azure AI Foundry the action (annotate or block) is set per guardrail, and document attack detection is off by default.

Detection techniques: rules, classifiers and LLM judges

Rules and patterns

Regular expressions, word lists, length caps, URL allowlists and schema validation are cheap, fast and explainable (Bedrock's word filters use "case-insensitive whole-word and whole-phrase matching"). They suit secrets, card numbers and output formats, not intent: rephrasing, translation and encoding defeat keyword lists.

Purpose-built classifiers

Meta's Llama Guard 3 is a fine-tuned Llama 3.1 8B model that classifies both prompts and responses against 14 categories (S1 to S14): 13 hazards based on the MLCommons taxonomy plus S14 Code Interpreter Abuse for tool use. Its unsafe score is the probability of the first output token, so the threshold is yours to set. Azure Prompt Shields separates user prompt attacks from document attacks hidden in emails, files and web pages. Both vendors state limits: Llama Guard "may be susceptible to adversarial attacks or prompt injection attacks", and Microsoft says Prompt Shields models are "primarily trained and evaluated on English content".

Secondary LLM call as a guardrail

A second, usually smaller, model judges the input or draft output against a rubric. The OpenAI cookbook example scores output from 1 to 5 and blocks "anything scoring a 3 or higher", and suggests sending guardrails "asynchronously along with your main LLM call" to hide latency. It also warns that an LLM guardrail has "the same vulnerabilities as your base LLM call itself", so one injection can evade both, and recommends combining it with rules or traditional ML.

PII redaction and data leakage controls

NIST AI 600-1 suggests organisations "leverage approaches to detect the presence of PII or sensitive data in generated output" (MP-4.1-009). Guardrails apply it both ways: data users send to a model, and data the model returns.

• Block, mask or tokenise: block secrets and regulated identifiers; mask (placeholder) to keep the conversation usable; use reversible tokenisation when an approved downstream step must restore the value. NeMo Guardrails and Bedrock both support block or mask on input and output.
• Detection is probabilistic: AWS describes its sensitive information filter as "probabilistic detection" for entities such as addresses and dates of birth, with custom regex for your own patterns. Pair named-entity detection with regex for anything with a fixed format.
• Redact before you log: guardrail and gateway logs otherwise become a new store of the data you are protecting.
• Redaction is not access control: if retrieval returns another customer's record, filter retrieval by the caller's permissions rather than masking the output.
• Cover reasoning and tool traffic: Bedrock notes that guardrails evaluate inputs and responses "excluding reasoning content blocks". Reasoning text, tool arguments, images and attachments are common gaps.

AI firewalls against exfiltration and model theft

For self-hosted or fine-tuned models, the model is itself an asset. MITRE ATLAS (technique IDs taken from the ATLAS data published on GitHub) lists AML.T0024 Exfiltration via AI Inference API, including AML.T0024.002 Extract AI Model, "extract a functional copy of a private model", reached through AML.T0040 AI Model Inference API Access. Content classifiers miss this because each query looks benign; the controls are gateway functions: authenticated callers, per-identity quotas, alerting on high-volume or systematic query patterns, and returning only what the task needs. The joint NSA, CISA, FBI and Five Eyes guidance Deploying AI Systems Securely advises that APIs return the minimum data needed "to inhibit model inversion" and recommends watching for high-frequency or repetitive inputs.

Guardrails for tool calls and agent actions

Once a model can call tools, content filtering becomes secondary. OWASP LLM06:2025 Excessive Agency names the root causes as excessive functionality, permissions and autonomy, and its complete mediation advice is direct: "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not." An execution guardrail is the policy enforcement point that applies that rule to every proposed call.

• Allowlist and schema: only registered tools, with typed arguments validated in code. OWASP prefers a narrow tool (write a file) to an open-ended one (run a shell command).
• Policy check per call: evaluate tool, arguments, target, data classification and destination before execution; deny by default for deletes, payments, permission changes and outbound messages.
• Screen tool output as input: results from web, email and API tools are untrusted content. Azure Foundry can scan tool responses for document attacks (preview).
• Approval gates sized to impact: human approval for irreversible or external actions, showing the exact arguments; too many prompts and approvers stop reading.

Where identity fits

A tool-call guardrail can only decide correctly if it knows who is acting. Each agent needs its own identity, separate from the human it acts for, and the effective permission should be the intersection of the agent's and the delegated user's; OWASP recommends acting in the user's context with the narrowest OAuth scope. Issue short-lived, scoped tokens per task after the policy decision, so a bypassed classifier still meets an authorisation check. The PocketOS database deletion shows the cost of an agent holding a broad token with no action-level check. See the AI Agent Authorisation Guide and the Agentic AI Security Guide; for tool servers, the MCP Security Guide.

The limits of guardrails

OWASP states that "it is unclear if there are fool-proof methods of prevention for prompt injection", and NIST AI 600-1 asks organisations to "regularly evaluate GAI system vulnerabilities to possible circumvention of safety measures" (MS-2.6-007). Plan for misses.

Choosing an approach

• One application, one provider, no tools: provider or platform filters plus output schema validation and logging; a dedicated AI firewall adds little yet.
• Several teams or models: a central gateway owning keys, quotas, logging and baseline content and PII policy; applications add context-specific rails in the SDK.
• Agents with write access: execution guardrails and per-agent identity come first; content classifiers are a supplement. Budget for approval workflows.
• Regulated data: PII detection both ways, masked logs, residency of the guardrail service itself (it sees every prompt), and evidence of guardrail versions.
• Build or buy: open-source frameworks give control but need people to tune and patch them; managed services give maintained classifiers but tie policy to a vendor's taxonomy. The AI Security Platform Buyer's Guide covers evaluation criteria.
• Small vs large organisations: small teams should favour managed filters and deterministic controls; large organisations need a platform team owning guardrail policy as code and metrics across business units.

Implementing guardrails in phases

1. Inventory and route: list AI applications, models, tools and data sources; route model traffic through a gateway or shared SDK and block direct provider egress where you can.
2. Write the policy first: define per application the allowed topics, data classes, refusals (NIST AI 600-1 GV-3.2-003 wants criteria for "the kinds of queries GAI applications should refuse to respond to"), tools and approval thresholds, in version control.
3. Deploy deterministic controls: size and rate limits, Unicode normalisation, output schemas, URL allowlists, secret scanning and tool allowlists. These rarely cause false positives.
4. Add classifiers in detect mode: run classifiers in shadow or annotate mode on live traffic for two to four weeks. Lakera's Detect mode always returns flagged false; Guardrails AI's NOOP action only records the failure "in the logs".
5. Set thresholds from data: label a sample, build a confusion matrix per category, and set thresholds by risk: stricter for jailbreaks on agents with tools than for topic restriction on an FAQ bot.
6. Enforce in rings: block for internal users, then a pilot segment, then everyone, with a per-policy kill switch back to detect mode; NIST AI 600-1 GV-1.7-001 expects the ability to deactivate a GAI system when necessary.
7. Retest on every change: rerun the evaluation suite after any model, prompt, guardrail version or fine-tune change. NIST AI 600-1 MS-2.7-008: "Verify fine-tuning does not compromise safety and security controls."

Testing and red teaming guardrails

NIST AI 600-1 asks organisations to "regularly review security and safety guardrails" (MS-2.5-006) and to red team against attacks such as prompt injection (MS-2.7-007). Test the guardrail as a component and the application as a whole, and label attacks with NIST AI 100-2 E2025 terminology so results compare across tools.

• Two test sets: an attack set (jailbreak families, injection in documents, encoded and translated variants, multi-turn escalation, PII exfiltration) and a benign set from real, anonymised traffic, including edge cases such as staff discussing attacks.
• Repeat runs: outputs vary, so run each case several times, per language, modality and streaming path, and report rates with denominators.
• Test the action layer: assume the classifier is bypassed and confirm the tool policy, scopes and approvals still stop the harmful action. See Red Teaming AI Agents for Identity Abuse.

Detection and measurement

The joint Deploying AI Systems Securely guidance recommends collecting logs of inputs, outputs, intermediate states and errors with automated alerts. Per request, log caller and agent identity, application, model version, guardrail policy version, each check's verdict, score and latency, the action taken, and tool calls with arguments and decisions.

• Attack success rate: share of red team cases producing the harmful outcome, per category and release.
• False positive rate: share of benign evaluation cases blocked, plus user-reported wrong blocks per thousand sessions.
• Guardrail latency: p50 and p95 added latency per check; regressions are why teams quietly disable rails.
• Coverage: percentage of model traffic and tool calls that pass through a guardrail, and number of AI applications calling providers directly (target zero).
• Tool-call denials: per agent identity; a spike suggests injection or misconfiguration.
• Block rate trend: a falling block rate with steady traffic can signal a new bypass rather than safer users.

Framework mapping

Common failure patterns

• Treating the system prompt as the guardrail. Prompt instructions are advice, not enforcement. The Chevrolet dealer chatbot and the DPD chatbot incident show public bots talked out of their brief with no output or topic check behind them.
• Screening the user but not the data. Input classifiers on the chat box miss instructions arriving in email, documents and tool output. In EchoLeak, the researchers reported that the attack email evaded Microsoft's cross-prompt injection classifiers and its link redaction.
• A classifier in front of an over-privileged agent. When a filter is the only control between a model and a broad token, one bypass becomes an incident. The Meta AI Instagram account takeover shows an AI assistant performing a sensitive account action for the wrong person.
• Vendor thresholds left at default. Defaults suit someone else's traffic: silent misses, or so many false positives that the rail is switched off.
• Logs that recreate the leak. Gateways storing unmasked prompts and responses that many people can read.
• No retest after model changes. A new model version or fine-tune changes both base and judge behaviour; last quarter's green suite proves nothing.

Practitioner checklist

• Route model and tool traffic through a gateway or shared SDK, with a per-application guardrail policy in version control.
• Deploy deterministic controls first: limits, normalisation, schemas, URL allowlists, secret scanning, tool allowlists.
• Run classifiers in detect mode on live traffic before blocking, and keep a per-policy kill switch.
• Give every agent its own identity and enforce tool authorisation outside the model, using the intersection of agent and user permissions.
• Maintain attack and benign evaluation sets per language and modality; rerun them on every model, prompt or guardrail change.
• Set thresholds from a confusion matrix per category, not vendor defaults.
• Track attack success rate, false positive rate, p95 added latency, coverage and tool-call denials monthly.
• Turn each confirmed bypass into a regression test before closing the incident.

Standards and references

• OWASP: LLM01:2025 Prompt Injection
• OWASP: LLM06:2025 Excessive Agency
• NIST: AI 600-1 Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• NIST: AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• MITRE: ATLAS data
• NSA, CISA, FBI, NCSC-UK and partners: Deploying AI Systems Securely
• NVIDIA: NeMo Guardrails rail types
• Meta: Llama Guard 3 model card

Related NHI Mgmt Group resources: LLM Security Guide · Prompt Injection Guide · AI Agent Authorisation Guide · Agentic AI Security Guide · AI Security Platform Buyer's Guide