Inspect AI Guide: Safety and Security Evaluations

Security teams are asked to show that a model or agent resists prompt injection, refuses harmful requests and stays inside its sandbox, and ad hoc prompting in a chat window cannot produce evidence that is repeatable, scored and auditable. Inspect is an open-source framework for large language model evaluations, created by the UK AI Security Institute and developed with Meridian Labs, that turns those questions into versioned tasks with datasets, solvers and scorers. This guide explains how Inspect is built, which defender use cases it serves, how to run it safely when models execute code, and how its results feed hardening and assurance. It is written for AI security engineers, red teams and assurance leads who already understand why evaluations matter; the AI Security Evaluations Guide covers the discipline itself.

This is an independent guide from NHI Mgmt Group, based on Inspect's public documentation as of November 2026. It is not sponsored or reviewed by the UK AI Security Institute or Meridian Labs. Features change, so confirm settings against Inspect's current documentation.

Key takeaways

Three building blocks. A task combines a dataset of samples, a solver that produces answers (from a single generate() call to a react() agent loop) and a scorer that grades them.
Agents run tools inside sandboxes. Set sandbox="docker" on a task so model-generated code runs in a container; generated Compose configurations default to network_mode: none.
Host-side code is outside the sandbox. Custom tools, agents and scorers run on the host, so isolation depends on how you write and deploy them, not only on the container.
Approval policies and limits bound agent behaviour. Human or automated approvers can approve, modify, reject, escalate or terminate tool calls, and time, token, message, turn and cost limits stop runaway samples.
A large ready-made catalogue. The Inspect Evals library lists more than 180 evaluations, including cyber benchmarks such as Cybench, CyberSecEval and InterCode CTF, and agent safety sets such as AgentHarm and AgentDojo.

What the tool does

Inspect is a Python package, installed with pip install inspect-ai, released under the MIT licence from the UKGovernmentBEIS repository on GitHub. It provides built-in components for prompt engineering, tool use, multi-turn dialogue and model-graded evaluation, and it supports more than 20 model providers plus local inference through Hugging Face, vLLM and SGLang.

Core concepts

Component	What it does	Examples from the docs
Task	Binds dataset, solver and scorer; discovered by name through the `@task` decorator	`Task()` returned from a function decorated with `@task`
Dataset	Samples with `input` and `target` fields	`json_dataset()`, `hf_dataset()`, `FieldSpec` to map columns
Solver	Produces the model's answer or runs an agent	`generate()`, `react()`, external agents bridged into Inspect
Tools	Actions the model may call	`bash()`, Python, text editing, web search, computer use, custom tools, MCP tools
Scorer	Grades output against the target	`includes()`, `model_graded_qa()`, custom scorers
Sandbox	Isolated environment for tool execution	`docker` built in; `k8s`, Modal, Proxmox, Vagrant and others as extensions
Log viewer	Browse results and full transcripts	`inspect view`

Runs start from the command line with inspect eval <file.py> --model <provider/model> or from Python with eval(). Every run writes a log with each sample's messages, tool calls and scores, which is the evidence an assurance reviewer needs.

Defender use cases

Pre-deployment safety and security testing: run prompt injection, jailbreak and refusal suites against a candidate model or system prompt before release. Map results to ATLAS AML.T0051 LLM Prompt Injection and AML.T0054 LLM Jailbreak. See the OWASP LLM Top 10 LLM01 Prompt Injection Guide.
Agent safety: use agentic sets such as AgentHarm and AgentDojo to measure whether a tool-using agent follows injected instructions or carries out harmful multi-step tasks.
Cyber capability evaluation: benchmarks such as Cybench, the GDM Dangerous Capabilities Capture the Flag set, InterCode CTF and CyberSecEval measure how far a model can progress on offensive security tasks. Frontier labs and governments use these results to set safeguards; see the Frontier AI Model Risk Frameworks Guide.
Regression gates in CI: pin a dataset and scorer, run on every model, prompt or guardrail change, and fail the pipeline when the score drops below an agreed threshold.
Guardrail validation: run the same suite with and without an input or output filter to quantify what the control adds. See the AI Guardrails and AI Firewalls Guide.
Custom organisational tests: encode your own data leakage, tool-misuse and policy cases as samples, scored by text match or a grader model.

Deployment and safe use

Evaluations of agents and cyber capability deliberately ask a model to run code and attempt security tasks. Treat the evaluation environment as hostile and design it so a misbehaving agent cannot reach anything real.

Sandbox settings that matter

Setting	What the docs say	Our recommendation
`network_mode`	Generated Compose configs set `network_mode: none`. A custom `compose.yaml` or `ComposeConfig` replaces the generated config, so you must repeat it	Keep `none` unless the task needs a target network; when it does, use an internal project network with no route to the internet or production
Host mounts	Writable host mounts are blocked by default; `unsafe_allow_writable_host_mounts` overrides this	Leave the override off; pass inputs with `Sample.files` or Docker-managed volumes
Privileged access	Privileged containers, host devices and Docker sockets can still expose the host	Never mount the Docker socket or run privileged services in an evaluation image
Images	Docs recommend pinning image digests	Pin by digest and rebuild from a reviewed Dockerfile
Resources	`cpus` and `mem_limit` in compose; `max_sandboxes`, `max_subprocesses`, `max_samples` for concurrency	Cap CPU and memory per container and size concurrency to the host
Engine versions	Docker Engine 24.0.6 or later and Compose 2.21.0 or later (2.22.0 or later when pulling images from a registry); Inspect's checks cover feature support, and keeping the engine, runtime and host patched is the operator's job	Patch the host and container runtime on a schedule
`local` sandbox	No isolation; runs as the current user	Use only for tasks that execute no model-generated code

Rollout steps

Provision a dedicated host or cluster. Use a VM or Kubernetes namespace with no production network routes and no cloud instance role that can reach real resources.
Scope evaluation credentials. Model provider API keys are read from environment variables. Issue keys used only for evaluation, with spend limits, and store them in a secrets manager rather than shell profiles. See the Secrets Management Guide.
Start with question-answer tasks. Run tasks with no tools first to confirm datasets, scorers and logging.
Add tools inside docker sandboxes. Confirm network_mode: none in every custom compose file before enabling bash() or Python tools.
Apply an approval policy. Use --approval human during development, then a YAML policy that sends shell and Python calls to a human or a custom allowlist approver and auto-approves the rest. Remote MCP servers are blocked while a policy is active because their calls cannot be reviewed.
Set limits on every sample. Combine time_limit with working_limit, and add token_limit, message_limit or cost_limit. Exceeded sample limits end the sample early and it is still scored, usually as incorrect.
Review host-side code. Custom tools and scorers run outside the sandbox; review them like production code.
Clean up and roll back. Sandboxes are removed when a task ends; run inspect sandbox cleanup docker after interrupted runs, and use --no-sandbox-cleanup only when debugging.

Logs contain full prompts, model outputs and tool results. If datasets include sensitive or proprietary content, store logs with the same access controls and retention as the source data. See the AI Agent Sandboxing and Code Execution Guide for container isolation options.

Detection and hardening value

Inspect produces measurements; the value comes from acting on them. Use results to decide which controls to strengthen and to prove that changes did not regress safety.

Framework mapping

Framework	Reference	How Inspect evidence supports it
NIST SP 800-53 Rev. 5	SA-11 Developer Testing and Evaluation	Versioned tasks and logs document testing of AI components before release
NIST SP 800-53 Rev. 5	CA-8 Penetration Testing	Agentic and prompt injection suites act as repeatable adversarial testing
NIST CSF 2.0	ID.RA-01	Failing samples identify, validate and record weaknesses in a model or agent
MITRE ATLAS	AML.M0008 Validate AI Model	Scored evaluation is a direct implementation of model validation
MITRE ATLAS	AML.T0051, AML.T0054	Injection and jailbreak datasets test resistance to these techniques

Metrics to track

Attack success rate: share of injection or jailbreak samples where the model complied, per release.
Over-refusal rate: share of benign samples refused, so safety gains do not break usefulness.
Agent containment events: tool calls rejected by approvers or samples terminated, per run.
Regression delta: score change between the current and previous model, prompt or guardrail version.
Coverage: share of in-scope risks (for example, the OWASP LLM Top 10 categories you have adopted) with at least one maintained task.

Where identity fits

Agent evaluations exercise the same non-human identity risks you face in production: an agent with a token can call any API the token allows. Run agent tasks with credentials issued only for the evaluation, scoped to the sandbox targets, and revoked after the run. Results that show an agent misusing tools are a direct input to least-privilege design for production agent identities. See the Agent Identity and Access Guide.

Alternatives

Inspect is one way to run evaluations. Other approaches serve related needs, and many teams combine them:

Vulnerability scanners for LLMs: probe-based tools that send large libraries of known attack prompts and report failures. See the AI Red Teaming Guide.
Prompt and application test harnesses: configuration-driven test runners aimed at application teams that test prompts and retrieval pipelines in CI.
Red team orchestration frameworks: libraries that automate multi-turn attacker strategies against a target system.
Classical adversarial ML libraries: for image, tabular and audio models, libraries such as the Adversarial Robustness Toolbox test evasion, poisoning, extraction and inference. See the Adversarial Robustness Toolbox Guide.
Commercial testing platforms: managed services for organisations that prefer not to run their own infrastructure. The AI Red Teaming and Testing Tools Buyer's Guide sets out evaluation criteria.

Choose based on who will maintain the tests, whether you need agentic sandboxes, and whether results must be shared with regulators or external evaluators who already use a common format.

Practitioner checklist

Install Inspect in an isolated environment on a host with no route to production.
Issue evaluation-only model API keys with spend limits, held in a secrets manager.
Repeat network_mode: none in every custom compose file and pin images by digest.
Never mount the Docker socket, run privileged containers or enable writable host mounts.
Apply an approval policy for shell and Python tools and set time, working time and token limits on every task.
Review custom tools and scorers as production code, because they run on the host.
Start from relevant Inspect Evals sets, then add organisation-specific samples.
Gate model, prompt and guardrail changes in CI on attack success and over-refusal thresholds.
Store logs with access controls that match the sensitivity of the datasets.

Inspect AI Guide: Running Safety and Security Evaluations