Deterministic vs Probabilistic Controls in AI Systems: A Risk by Risk Decision Matrix

Most AI security programmes rely on controls that estimate risk, such as classifiers and LLM judges, for problems that need controls that enforce limits, such as authorisation, allowlists and schemas, and the gap only shows when an attacker adapts. A deterministic control gives the same result for the same input every time and can be tested to a pass or fail. A probabilistic control returns a likelihood and has a miss rate that moves with model versions, phrasing and attacker effort. Both have a place, but they are not interchangeable. This guide gives a risk by risk decision matrix for AI applications and agents: which risks must have a deterministic control as the boundary, which can only be handled probabilistically, and the layering rules that connect the two. It is for security architects, AI platform teams and risk owners who sign off AI deployments. Tool-call authorisation in depth is covered in Prompt Guardrails vs Policy Engines for AI Agent Actions; this page covers the whole risk surface.

Key takeaways

• Ask whether the risk can be stated as a rule. If the unacceptable outcome can be expressed in terms a program can check (this identity, this resource, this destination, this schema, this amount), enforce it deterministically. If it depends on meaning (is this text harmful, is this answer true), only a probabilistic control can judge it.
• Published bypass rates rule out classifiers as boundaries. An academic study of 12 recent defences reported adaptive attacks succeeding more than 90% of the time against most of them, and NIST measured agent hijacking success rising from 11% to 81% with new attacks.
• Regulators and standards bodies agree. The UK NCSC advises focusing on deterministic, non-LLM safeguards that constrain what the system can do, and OWASP says input screening is not a security boundary.
• Deterministic controls bound impact; probabilistic controls reduce frequency. Design so that the worst case after a missed detection is still acceptable, then use detection to make that worst case rare.
• Probabilistic controls can tighten, never loosen. A classifier score may move a request from allow to require approval or deny; it must never move a request from deny to allow.

The short answer

For every risk on your AI system's register, decide first whether a deterministic control can bound it. Tool permissions, data access, network egress, code execution, spending, output formats consumed by other systems, and approvals for irreversible actions all can, so they need hard limits that hold even when the model is fully controlled by an attacker. Harmful content, factual accuracy, tone, and detection of injection attempts cannot be fully stated as rules, so they rely on classifiers, LLM judges and human review, measured as rates. Layer them so the deterministic control is the floor and the probabilistic control is an early warning and a tightening signal.

Two kinds of control compared

The categories describe how a control decides, not where it runs. A regular expression that blocks a known secret format is deterministic; a model-based PII detector inside the same gateway is probabilistic. Many AI firewall products combine both, so classify each rule you rely on, not the product. Techniques and placement are covered in the AI Guardrails and AI Firewalls Guide.

Why probabilistic controls cannot be the boundary: bypass evidence

The argument is empirical, not philosophical. Three sources set the baseline for any risk acceptance decision.

• Adaptive attacks: The Attacker Moves Second (Nasr, Carlini, Tramèr and others, October 2025) bypassed 12 recent defences against jailbreaks and prompt injection using gradient descent, reinforcement learning, random search and human-guided exploration. Attack success exceeded 90% for most defences, most of which had originally reported near-zero attack success. The lesson for buyers: a vendor's detection rate measured against a fixed test set says little about a motivated attacker.
• Agent hijacking: NIST's evaluation of agent hijacking (January 2025) found that against one model the strongest new attack succeeded 81% of the time, against 11% for the strongest baseline attack, and that repeating attempts raised average success across five injection tasks from 57% on one attempt to 80% after 25 attempts. Attackers retry; a per-attempt miss rate compounds.
• Standards view: OWASP's Top 10 for LLM Applications 2025 states it is unclear whether fool-proof methods of preventing prompt injection exist, and its Prompt Injection Prevention Cheat Sheet notes that a guardrail LLM "is itself susceptible to prompt injection". The NCSC writes that prompt injection may never be totally mitigated in the way SQL injection can be.

None of this makes detection worthless. A classifier that stops most unsophisticated attempts reduces noise, load on reviewers and opportunistic abuse. It means the residual risk after detection must be acceptable on its own, which only a deterministic limit can guarantee.

The decision matrix: which control owns each risk

Each row names the risk, the deterministic control that should bound it, the probabilistic control that adds value, and which one is the boundary. OWASP LLM Top 10 2025 IDs are shown where the risk maps to one.

The pattern is clear: everything about what the system can touch, reach, run or spend is deterministic; everything about what text means is probabilistic. Prompt injection sits in between: you cannot reliably detect it, so you limit what a successful injection can achieve. The Secure Agent Design Patterns Guide covers architectures, such as plan-then-execute and dual LLM designs, that make the deterministic rows easier to enforce; the 2025 paper Design Patterns for Securing LLM Agents against Prompt Injections frames these patterns as a route to provable resistance by constraining what the agent can do.

Where hard limits are required

Four areas need deterministic enforcement at a point the model cannot modify or route around, because a single miss is not recoverable.

Tool permissions

The agent's identity holds only the tools and scopes its task needs, and every call passes through an enforcement point that checks policy before execution. OWASP's AI Agent Security Cheat Sheet asks builders to grant the minimum tools required, fail closed for unknown tools, and not to rely solely on model output for authorisation decisions; it also notes that a risk classification "does not grant permission to run a tool". Design and testing are in the AI Agent Authorisation Guide and Tool-Call Security for AI Agents.

Egress

Exfiltration needs a channel out. In the EchoLeak case, a crafted email evaded the injection classifier, reference-style links and images survived redaction, and trusted-domain services allowed by the Content Security Policy fetched attacker-supplied URLs carrying data. The vendor fixed it server side and reported no exploitation. The lesson is that the last line was an allowlist that was broader than it needed to be; a narrower deterministic allowlist would have held after both probabilistic and pattern defences failed. See the Agent Egress Control and Network Isolation Guide.

Code execution

If an agent can run code, assume it will run attacker-chosen code. OWASP lists executing arbitrary code without sandboxing as a practice to avoid. The sandbox, not a code reviewer model, is the boundary: no credentials, no network unless allowlisted, file system and time limits. See the AI Agent Sandboxing and Code Execution Guide.

Spending and irreversible actions

Payments, deletions, permission grants and production deployments need caps and approvals enforced outside the model. A prompt asking for human review does not itself stop execution, as the OWASP cheat sheet notes for agent frameworks. The approval must be bound to the specific action and checked at execution. See the AI Agent Action Approvals and Spending Limits Guide.

Layering rules

1. Deterministic first, on the only path: every action that touches a resource passes a deterministic check at an enforcement point outside the agent's process. If there is a second path (raw shell, direct API credentials), the check is decorative.
2. Probabilistic signals only tighten: a high risk score can deny, require approval or lower a rate limit. A "safe" verdict never grants anything the deterministic policy would not.
3. Attributes come from trusted sources: deterministic rules evaluate identity, resource and approval data from tokens, inventories and approval services, never from text the model produced.
4. Taint follows untrusted input: once an agent has read untrusted content, its effective privileges fall to the trust level of that content. NCSC cites this principle; deterministic rules can apply it by removing high-risk tools from sessions that have ingested external email or web content.
5. Decide fail-open or fail-closed per control: deterministic controls on write, delete, payment and egress fail closed; a probabilistic content filter on a low-risk read path may fail open with logging, if that is a written decision.
6. Measure each layer separately: record which layer acted on every event, so you can see how much the deterministic layer is carrying when the probabilistic one misses.

Decision guidance

• Customer-facing chat with no tools: probabilistic controls carry most of the load (content safety, groundedness), with deterministic limits on retrieval scope, rate and cost. Accept that some harmful outputs will occur and plan review and response.
• Internal agent with read tools over sensitive data: permission-aware retrieval and egress allowlists are mandatory; classifiers add triage.
• Agent with write, payment or admin tools: deterministic authorisation, approvals and caps for every write; probabilistic checks only route more actions to approval.
• Agent that reads untrusted external content and holds private data: remove either the untrusted input, the sensitive access or the outbound channel by design; detection alone is not an acceptable control here.
• Small organisations: start with deny-by-default tool allowlists, a narrow egress allowlist and spending caps, which cost little; buy or build classifiers after those are in place.
• Large organisations: centralise deterministic policy in a gateway and policy engine with one decision log; run probabilistic controls as shared services with published, measured miss rates. See the AI Security Architecture Reference Guide for where each sits.

Worked scenario: an inbox assistant

A firm deploys an assistant that reads staff email, searches internal documents and can draft replies, create calendar entries and send messages. The risk owner is told an AI firewall detects 99% of injection attempts.

1. Threat: an external sender includes hidden instructions to forward recent board papers to an outside address. External email is untrusted input processed by an agent that holds sensitive data and has an outbound channel.
2. Probabilistic only: the firewall catches the first wording. The attacker rephrases and retries; given the published adaptive results, assume a later attempt passes. The agent sends the papers.
3. Deterministic redesign: sending to external recipients requires approval bound to the recipient and attachment; document search is filtered to the user's own entitlements; rendered links and images are restricted to an allowlist of internal domains; sessions that have read external email lose the send tool until a human confirms.
4. Layered result: the firewall still flags most attempts and raises a risk score, which also routes internal sends to approval for that session. When an attempt is missed, the send is held for approval and the user sees an unexpected external recipient with board papers attached, and declines.
5. Evidence: the log shows the classifier verdict, the policy decision, the approval request and the decline on one record, which is what the risk owner needed to accept the residual risk.

Implementation roadmap

1. Build the risk register per AI system: list the rows of the matrix that apply, the current control for each, and whether it is deterministic or probabilistic.
2. Find probabilistic-only boundaries: any row where the only control is a classifier, judge or system prompt instruction for a risk the matrix marks deterministic is a priority gap. OWASP LLM07 is explicit that the system prompt should not be used as a security control.
3. Add deterministic controls in audit mode: deploy tool policies, egress allowlists and schemas logging would-deny results for two to four weeks; compare with real traffic.
4. Enforce by risk tier: start with payment, delete, admin and outbound tools; keep a per-rule switch so one rule can be rolled back without disabling enforcement.
5. Re-tune probabilistic controls as signals: wire their outputs into the policy as risk attributes and set thresholds against benign traffic.
6. Test adaptively: red team with retries and rephrasing, not only fixed test sets, and record miss rates per control. The AI Red Team Test Plan Template helps structure this.

Detection and measurement

• Boundary coverage: percentage of matrix rows for each system with a deterministic boundary in place, target 100% for rows marked deterministic.
• Carry rate: in red team runs, the share of attempts the probabilistic layer missed that the deterministic layer stopped. A falling figure means detection improved; a rising figure means the hard limits are doing the work.
• Escape rate: attempts that passed both layers. Any non-zero result is a design gap to fix, not a threshold to tune.
• Adaptive miss rate per classifier: measured with retries, reported with the test date and model version.
• Policy bypass signals: downstream calls made with agent credentials and no matching policy decision record.
• Override and approval rates: near-100% approval of held actions suggests reviewers are rubber-stamping and the gate is weakening.

Framework mapping

Common failure patterns

• Vendor detection rate used as risk acceptance: a 99% figure from a static test set approved as if it were a boundary, with no hard limit behind it.
• Instructions as controls: "never send data externally" in the system prompt, with the send tool and open egress still available.
• Allowlists wider than the use case: trusted-domain lists that include services able to fetch arbitrary URLs, which turns the allowlist into an exfiltration path, as EchoLeak showed.
• Classifier verdict as permission: code that executes a tool call when the guardrail returns "safe", inverting the layering rule.
• LLM judge guarding an LLM: a second model used as the only check on the first, exposed to the same injected content.
• Measuring only the detector: dashboards report blocked attempts but not escapes, so nobody knows whether hard limits exist where they matter.

Practitioner checklist

• Every AI system has a risk register with each risk labelled deterministic, probabilistic or shared.
• Tool access is deny by default, per agent, checked at an enforcement point outside the agent.
• Retrieval is filtered on the end user's entitlements before content reaches the model.
• Egress, rendered links and images are restricted to explicit allowlists.
• Code runs only in sandboxes without credentials or open network access.
• Payments, deletions, permission changes and external sends have caps and bound approvals.
• Model output consumed by other systems is validated against strict schemas.
• Classifier and judge outputs can only tighten decisions, never grant.
• Red team tests use adaptive attackers with retries; miss and escape rates are reported per layer.

Standards and references

• OWASP: Top 10 for LLM Applications 2025
• OWASP: LLM Prompt Injection Prevention Cheat Sheet
• OWASP: AI Agent Security Cheat Sheet
• NCSC: Prompt injection is not SQL injection
• NIST: Technical Blog, Strengthening AI Agent Hijacking Evaluations
• arXiv: The Attacker Moves Second, Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections
• arXiv: Design Patterns for Securing LLM Agents against Prompt Injections
• MITRE ATLAS: Exfiltration via AI Agent Tool Invocation

Related NHI Mgmt Group resources: Prompt Guardrails vs Policy Engines for AI Agent Actions · AI Guardrails and AI Firewalls Guide · Secure Agent Design Patterns Guide · Prompt Injection Guide