AI Control Guide: Trusted Monitoring and Control Protocols

Most AI agent security assumes the model is on your side and the attacker is outside, but nobody can yet prove that a capable model will never pursue a goal of its own. AI control is the research field that asks a harder question: can we deploy a model safely even if it is actively trying to subvert our safeguards? It answers with control protocols (monitoring, deferral, editing and resampling built from a weaker trusted model and a small budget of human review) and with control evaluations, where a red team plays the misaligned model. This guide explains the vocabulary, the main protocols and what the published evidence shows, then turns it into an enterprise design for agent deployments: choke points, monitors, audit queues, scoped identities and metrics. It is for security architects, AI platform teams and red teams running agents with real permissions.

Key takeaways

Control assumes the worst about the model. Alignment work tries to make the model want the right thing; control asks whether the surrounding system still prevents unacceptable outcomes if it does not.
Protocols combine a strong untrusted model, a weaker trusted one and scarce humans. The trusted model scores every action for suspicion, the most suspicious go to human audit, and the rest are passed, edited, resampled or replaced.
The evidence is promising but narrow. In the Ctrl-Z agent study, resampling plus analysis of earlier steps cut attack success from 58% to 7% at about a 5% usefulness cost, on one benchmark of 257 system administration tasks.
A monitor only sees what passes through a choke point. Control depends on routing every tool call through a gateway the agent cannot bypass, with credentials scoped so the worst single action is survivable.
Measure safety and usefulness together. A protocol that blocks everything is safe and useless; control evaluations report both, with a red team that moves last.

What AI control is, and what it is not

The term comes from Greenblatt, Shlegeris, Sachan and Roger's paper "AI Control: Improving Safety Despite Intentional Subversion" (first posted December 2023). Its setting was simple: solve programming problems with a powerful but untrusted model (GPT-4), a less capable trusted model (GPT-3.5) and limited access to trusted, high-quality human labour, without ever submitting code containing a backdoor, defined as a logic error that the test cases do not catch. The authors found that protocols such as trusted editing and untrusted monitoring improved substantially on simple baselines.

Approach	Question it answers	Assumed adversary	Typical controls
Alignment and safety training	Does the model want to behave well?	None; the model is the thing being shaped	Training, fine-tuning, constitutions, evaluations of propensity
Guardrails and AI firewalls	Can we block harmful inputs and outputs?	External users and injected content	Input and output classifiers, content filters, policy engines
AI control	Are we safe even if the model is trying to subvert us?	The model itself, played by a red team	Trusted monitoring, deferral, resampling, human audit, shutdown, least privilege

The approaches stack. Guardrails stop a prompt-injected agent; control assumes the agent may already be compromised, by injection or by its own goals, and limits what it can do anyway. See the AI Guardrails and AI Firewalls Guide and the Agent Controls and Monitoring Guide.

Core vocabulary

The definitions below follow the UK AI Security Institute's ControlArena documentation, which was built with Redwood Research.

Term	Meaning	Enterprise equivalent
Untrusted model	The capable model used for its abilities, with no assumption about its goals or honesty	The frontier model driving your agent
Trusted model	A weaker model very unlikely to cause harm on purpose, trusted because of testing, limited capability or architecture	A smaller model, classifier or rules engine used as the monitor
Human auditing budget	A small supply of expert time spent on the most suspicious outputs	The analyst hours your SOC or platform team can give the review queue
Honest and attack policies	The model behaving normally, and a red-team version that also pursues a hidden side task	Normal agent runs, and seeded red-team runs
Safety	One minus the probability of a catastrophic outcome, estimated with the attack policy	Share of seeded attacks stopped before harm
Usefulness	Share of intended tasks completed, estimated with the honest policy	Task success rate with controls on, compared with off
Micro-protocol	Per-task or per-action strategy that uses monitor scores	Gateway rules on each tool call
Macro-protocol	Deployment-wide strategy: when to escalate, when to shut the model down, how to adapt to accumulated evidence	Incident thresholds, kill switch and model rollback decisions

The main control protocols

Protocol	How it works	Strength	Weakness or cost
Trusted monitoring	The trusted model reviews every untrusted output and assigns a suspicion score; the top scores go to human audit	Simple; works with any agent; produces an audit trail	A weaker monitor can miss subtle attacks; audit capacity caps how many flags get human review
Defer to trusted	Suspicious untrusted outputs are replaced by the trusted model's own output	Removes the suspicious action entirely	Lower quality on deferred tasks, so usefulness drops as the threshold tightens
Trusted editing	The trusted model edits out the suspicious parts of the untrusted output	Keeps most of the capable model's work	Edits can miss the harmful part or break the result
Defer to resample	On a suspicious action, draw several new samples from the untrusted model and pick the least suspicious	Keeps capability; repeated attack attempts become evidence against the model	Extra inference cost; a model that attacks consistently still needs deferral
Untrusted monitoring	A separate instance of the capable model acts as monitor	Catches subtler attacks than a weak monitor	Collusion: the monitor may overlook its twin; mitigated by redacting details, different prompts and several monitor instances

Two practical points follow. First, threshold choice is an economic decision: set it so the number of flagged actions matches the hours you can actually audit. Second, protocols operate on actions, not conversations, so they work best when the agent's effects are expressed as discrete tool calls a gateway can hold.

What the evidence shows so far

Code backdoors (2023 to 2024): the original paper showed that trusted editing and untrusted monitoring beat simple baselines at stopping backdoored code from a deliberately subversive GPT-4.
Agents in a shell (2025): "Ctrl-Z: Controlling AI Agents via Resampling" ran what its authors call the first control evaluation in an agent environment, using BashBench, 257 multi-step system administration tasks in which the adversarial agent tries to download and run malicious code secretly. The best protocol, resampling combined with analysis of earlier steps, cut attack success from 58% to 7% for about a 5% loss of usefulness.
Safety cases (2025): "A sketch of an AI control safety case" argues that a developer could show an internal LLM agent will not exfiltrate sensitive data, using a red team that builds models to try, in a proxy of the real environment. The argument rests on three claims: the red team elicits the model's real capability, controls are at least as effective in deployment as in testing, and results are extrapolated conservatively.
Chain-of-thought monitoring (2025): a multi-organisation position paper argues that reading a model's reasoning in natural language is a useful but fragile oversight signal that development choices could erode, and that, like all known oversight, it misses some misbehaviour.
Frontier lab policy: Google DeepMind's Frontier Safety Framework version 3 (September 2025, updated to 3.1 in April 2026) added misalignment scenarios in which models interfere with operators' ability to direct, modify or shut them down. See the Frontier AI Model Risk Frameworks Guide.

Treat these numbers as research results on specific benchmarks, not as guarantees for your deployment. The value for enterprises is the method: a defined threat model, a measured protocol and an adversarial test.

Applying control to enterprise agent deployments

Most organisations consume frontier models through an API and cannot train a deliberately misaligned model. They can still apply the structure. The mapping below is NHI Mgmt Group's synthesis of the research for practitioners.

Define unacceptable outcomes first: a short list such as production data deletion, exfiltration of customer data or credentials, changes to IAM policy, payments, and disabling logging. These are the side tasks a red team will attempt.
Build one choke point: route every tool call, API request and code execution through a gateway that can hold, score, edit or deny it. An agent with a direct network path or raw cloud credentials cannot be controlled. See the Agent Egress Control and Network Isolation Guide.
Separate the monitor: run the monitor as a different model or classifier, with its own identity and credentials, outside anything the agent can modify. The agent must not be able to edit its own monitor, prompts or logs (separation of duties).
Tier actions by stakes: read-only actions pass with logging; reversible writes are scored and resampled when suspicious; irreversible or high-impact actions always need human approval. See Dual control and four-eyes approval for privileged and agent actions.
Look at trajectories, not single steps: the Ctrl-Z result came from combining per-action resampling with analysis of earlier steps; an exfiltration can be split into innocent-looking pieces.
Write the macro-protocol: how many confirmed suspicious actions trigger suspension of the agent, a model rollback or an incident. See the AI Agent Kill Switch and Containment Guide.

Where identity fits

Control protocols decide whether an action should run; identity decides what an action can reach if it does. A monitor that misses one action in a hundred is acceptable when the agent's credentials cannot delete production, and catastrophic when they can, as the PocketOS database deletion showed when an AI coding agent used an over-privileged platform token to wipe production. Give each agent instance its own short-lived identity, scope tokens to the task, keep high-impact permissions behind just-in-time approval, and make sure the monitor, the gateway and the audit log run under identities the agent cannot assume. See the Agentic AI Identity Guide.

How to implement

Inventory agents and permissions: list each agent, its model, its tools, the credentials it holds and the unacceptable outcomes those credentials make possible.
Insert the gateway in log-only mode: route all tool calls through it and score them with the trusted monitor, without blocking, for two to four weeks. This gives you the score distribution on honest traffic.
Set thresholds from audit capacity: choose the audit threshold so the expected daily flags fit the review hours you have; set a lower resample threshold and a higher automatic deny threshold.
Run a control evaluation in a proxy environment: give a red team the agent, the protocol and the side-task list, let it write attack prompts or scaffolds, and measure safety and usefulness. Isolate the proxy properly: the OpenAI and Hugging Face evaluation sandbox escape shows what happens when a test environment can reach real credentials.
Enforce for the highest-stakes tier first: turn on deny and human approval for irreversible actions, then resampling for reversible writes.
Repeat on every model change: a new model version is a new untrusted model; rerun the evaluation before promoting it.
Roll back safely: keep log-only mode as a switch per action tier, so a broken monitor degrades to logging plus human approval for high-stakes actions, never to unmonitored execution.

Decision guidance

Trusted monitoring only: enough for agents with read-mostly permissions and reversible actions, and for small teams with little audit capacity.
Add resampling: when agents perform many reversible write actions and deferral to a weaker model would hurt quality too much.
Add untrusted monitoring: when attacks are subtle enough that a small monitor misses them, and you can run a second capable instance with anti-collusion measures. It costs more inference.
Small organisations: use the provider's model as agent, a smaller model or classifier as monitor, a hard approval gate for irreversible actions and tight credential scoping. Skip bespoke evaluations at first; use prompted red-team runs.
Large organisations and AI developers: run formal control evaluations, ideally with open tooling such as ControlArena (UK AI Security Institute and Redwood Research), and keep the results as evidence for risk committees.

Detection and measurement

Seeded attack catch rate: share of red-team side-task attempts stopped before harm, per action tier (the practical safety estimate).
Usefulness cost: the drop in task success with the protocol on compared with it off; the Ctrl-Z result of about 5% is a useful reference point.
Audit queue precision and latency: share of flagged actions that reviewers judge genuinely suspicious, and time to decision.
Choke point coverage: share of agent actions that passed through the gateway; anything below 100% is a bypass to investigate.
Resample rate trend: a rise after a model update is an early sign of changed behaviour.
Privilege exposure: number of agent identities holding permissions that enable an unacceptable outcome without approval (target zero).

Common failure patterns

Monitoring conversations, not actions: the agent's text looks fine while a tool call does the damage.
Bypassable choke points: agents with shell access, raw cloud keys or unrestricted egress can act outside the gateway.
Thresholds that ignore audit capacity: queues of thousands of flags get rubber-stamped, which turns human audit into theatre.
Testing once: a control evaluation on last quarter's model says little about this quarter's.
Self-monitoring without anti-collusion measures: the same model, prompt and context as monitor and actor.
Overstating research numbers: quoting a benchmark result as the safety level of a different deployment.

Framework mapping

Framework	Reference	Control practice
NIST SP 800-53 Rev 5	SI-4 System Monitoring; AU-6 Audit Record Review, Analysis, and Reporting; AC-5 Separation of Duties; AC-6 Least Privilege; CA-8 Penetration Testing; SA-11 Developer Testing and Evaluation; IR-4 Incident Handling	Trusted monitoring, human audit queue, monitor isolated from agent, scoped credentials, control evaluations, macro-protocol shutdown
NIST CSF 2.0	ID.RA-05; DE.CM-09; DE.AE-02	Unacceptable-outcome analysis; runtime monitoring of agent actions; analysis of flagged events
MITRE ATLAS	AML.T0053 AI Agent Tool Invocation; AML.T0051 LLM Prompt Injection; AML.M0029 Human In-the-Loop for AI Agent Actions; AML.M0030 Restrict AI Agent Tool Invocation on Untrusted Data	Gating tool calls, human approval for high-stakes actions
OWASP Top 10 for LLM Applications 2025	LLM06:2025 Excessive Agency	Limiting functionality, permissions and autonomy so a subverted agent has little to misuse

Practitioner checklist

Write the list of unacceptable outcomes for each agent and review it with the business owner.
Route every agent tool call through a gateway the agent cannot bypass or modify.
Run a separate trusted monitor with its own identity and score every action.
Set audit thresholds from real reviewer capacity and record the reasoning.
Require human approval for irreversible and high-impact actions.
Give each agent instance short-lived, task-scoped credentials with no standing high-impact permissions.
Run a control evaluation in an isolated proxy environment before each model promotion.
Define the macro-protocol: suspension and rollback thresholds tied to the kill switch.
Report safety, usefulness, coverage and audit precision to the AI risk committee.

AI Control Guide: Monitoring and Containing Agents You Cannot Fully Trust