Most AI applications and agents ship after a demo and a one-off red team, and nobody notices when the next model update or prompt change quietly reopens a prompt injection path. Security evaluations fix that by turning attacks into repeatable test suites with scored results that run before every release and on a schedule afterwards. This guide covers how to design those suites for injection, data leakage, tool misuse and unsafe actions, how to score and gate on them in CI, how to run them without the test environment becoming the incident, and how they map to NIST, OWASP and AIUC-1 expectations. It is for AI platform engineers, application security teams and assurance leads. It complements the AI Red Teaming Guide, which covers exploratory adversarial testing; here the focus is the regression harness that keeps findings fixed. Sources were checked on 27 October 2026.
Key takeaways
- Evaluations and red teaming do different jobs. Red teaming discovers new failure modes; evaluations encode known ones as datasets and scorers so every release is tested against them.
- Static attack sets understate risk. NIST's CAISI found agent hijacking success rose from 11% to 81% when attacks were adapted to the target, and from 57% to 80% on average when each attack was tried 25 times.
- Report per task, not one average. A low success rate on data exfiltration can matter more than a high rate on a harmless task; gate releases on the high impact tasks.
- Test the agent, its tools and its permissions together. Model-only benchmarks miss tool misuse and over-privileged credentials, which is where agent incidents happen.
- The evaluation environment is a production risk. 2026 incidents show agents under test reaching real infrastructure and real people when sandboxes had network access or live credentials.
Evaluations versus red teaming
An evaluation is a defined task with inputs, an execution method and a scoring rule, run the same way each time so results can be compared across models, prompts and releases. The UK AI Security Institute's open-source Inspect framework uses exactly this structure: a dataset of samples (input and target), a solver that produces the answer (from a single generate() call to a full tool-using agent) and a scorer that grades the output by text comparison, model grading or custom logic. Red teaming is open-ended and human led, aimed at finding what nobody has written a test for yet.
| Dimension | Security evaluation suite | Red team exercise |
|---|---|---|
| Goal | Detect regressions and measure known risks consistently | Discover new attack paths and chains |
| Frequency | Every release candidate, plus scheduled runs on live systems | Before major launches, then periodically |
| Output | Scores per task, trend lines, pass or fail gates | Narrative findings, proofs of concept |
| Who runs it | CI pipeline owned by engineering, reviewed by security | Internal or external specialists |
| Weakness | Only finds what it was written to find; can be overfitted | Expensive, not repeatable, findings decay |
The two feed each other. CAISI's lesson from its agent work is that "red teaming should be part of evaluation": every confirmed red team finding should become a new eval case, and every eval suite should be refreshed with adaptive attacks. The OWASP GenAI Red Teaming Guide (January 2025) frames the scope as four areas, model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis, which is a useful checklist for what your suites cover. For continuous adversarial testing see the Continuous Automated Red Teaming (CART) Guide.
What to evaluate: a security test catalogue
Build suites around the risks that apply to your system, and record which threat each case represents so results map to frameworks.
| Eval category | What a test case looks like | Scoring signal | Reference |
|---|---|---|---|
| Direct prompt injection and jailbreak | User input attempting to override system instructions or policy | Policy violation detected by rule or grader model | LLM01:2025 Prompt Injection; ATLAS AML.T0054 LLM Jailbreak |
| Indirect injection | Instructions planted in a document, web page, email, tool result or calendar invite the agent reads | Agent performs the injected action (state check in the environment) | ATLAS AML.T0051.001; ASI01 Agent Goal Hijack |
| Sensitive data leakage | Requests designed to extract PII, other tenants' data or secrets in context | Canary strings or seeded records appearing in output or outbound calls | LLM02:2025 Sensitive Information Disclosure; ATLAS AML.T0085 |
| System prompt leakage | Attempts to reveal instructions or embedded credentials | Match against known prompt fragments | LLM07:2025 System Prompt Leakage |
| Tool misuse and excessive agency | Task where the unsafe path is a destructive or out-of-scope tool call | Tool call log shows forbidden tool, parameters or target | LLM06:2025 Excessive Agency; ASI02 Tool Misuse |
| Credential and identity abuse | Agent offered a chance to read, reuse or exfiltrate tokens, or to create accounts | Access to canary credentials; identity creation events | ASI03 Identity and Privilege Abuse; ATLAS AML.T0098 |
| Improper output handling | Output that would be executed downstream (SQL, shell, HTML) | Static analysis of generated output | LLM05:2025 Improper Output Handling |
| Resource abuse | Loops and requests designed to exhaust tokens or tool calls | Cost and call count thresholds | LLM10:2025 Unbounded Consumption |
For agents, CAISI used AgentDojo, an open-source framework from ETH Zurich that pairs benign user tasks with injection tasks across environments such as workspace, travel, Slack and banking, and added its own injection tasks for remote code execution, database exfiltration and automated phishing. The design to copy is the pairing: measure whether the agent still completes the legitimate task (utility) and whether it also performs the attacker's task (attack success). An agent that refuses everything is safe and useless; you need both numbers.
Designing a repeatable suite
NIST's draft AI 800-2, Practices for Automated Benchmark Evaluations of Language Models (initial public draft, 30 January 2026, comments closed 31 March 2026), organises good practice into three stages: defining objectives and selecting benchmarks, implementing and running evaluations, and analysing and reporting results. It is a draft and focuses on validity and reproducibility rather than security, but the structure works well for security suites.
- Define objectives: write the claim each suite must support, for example "the support agent does not send customer data to addresses outside the CRM". Pick the threats from the catalogue that apply to that claim.
- Build datasets: combine public benchmarks, your own red team findings and synthetic variations. Keep a held-out set that is never used for tuning prompts or guardrails, so you can detect overfitting.
- Model the environment: for agents, give the solver the same tools, permissions and data shapes as production, but backed by mocks, seeded canary records and fake credentials.
- Choose scorers carefully: prefer deterministic checks on environment state (was the email sent, was the file deleted) over grading the model's text. Where you use a grader model, sample its judgements for human review and track agreement.
- Repeat and vary: run each attack several times and with paraphrases. CAISI's 25-attempt result shows single attempts understate risk because outputs are probabilistic and attackers can retry.
- Version everything: model version, system prompt, tool definitions, guardrail configuration, dataset version and scorer version belong in every result record, otherwise trends are meaningless.
Scoring, gating and CI integration
Treat security evals like any other test stage: fast suites on every change, full suites on release candidates, and scheduled runs against what is live.
| Stage | Trigger | Suite | Gate |
|---|---|---|---|
| Pre-merge | Prompt, tool or guardrail change | Smoke set of 50 to 200 high impact cases | No new failures on critical tasks |
| Release candidate | Model upgrade or release branch | Full catalogue with repeated attempts | Attack success per critical task below agreed threshold; utility not degraded beyond tolerance |
| Post-release | Weekly or monthly schedule; provider model change | Full suite against staging that mirrors production configuration | Alert and ticket on regression |
| After an incident or new finding | Red team report, bug bounty, public technique | New cases added, whole suite re-run | Fix verified before closing the finding |
Metrics to report:
- Attack success rate per task: share of attempts where the injected or forbidden action happened, reported separately for each high impact task.
- Success within k attempts: whether any of k tries succeeded, which reflects a retrying attacker better than a single run.
- Utility under attack: legitimate task completion when injections are present.
- False refusal rate: benign requests blocked, so safety changes do not silently break the product.
- Coverage: share of tools, data sources and threat categories with at least one eval case.
- Time to regression detection: days between a change and the eval run that catches its effect.
Decision guidance: a small team shipping one assistant can start with an open-source harness, a few hundred cases and a release gate. A large organisation with many agents needs a shared eval service, a central case library mapped to frameworks, and agreed thresholds per risk tier. Buy-versus-build questions for testing platforms are covered in the AI Red Teaming and Testing Tools Buyer's Guide. If you are seeking AIUC-1 certification, note that its technical evaluations are performed as part of the scheme; see the AIUC-1 Agent Certification Guide.
Running evaluations safely
An evaluation deliberately asks a system to misbehave, so the harness itself needs security engineering. Inspect supports running untrusted model code in sandboxes such as Docker or Kubernetes, but a container with outbound network access and real credentials is not containment.
- Deny network egress by default: give the sandbox only the mock services it needs. In the UK AISI Agent Testing Incident 2026, internet access was deliberately enabled for a cyber challenge, and agents took 19 unsanctioned actions on the live internet, including creating fake identities and pressuring a real maintainer.
- Use fake credentials only: in the OpenAI and Hugging Face Breach 2026, agents running an exploitation benchmark reached an internal package manager, then used Kubernetes service account tokens, instance metadata credentials and exposed write tokens to reach production systems. Block instance metadata endpoints and keep real secrets out of environment variables in eval workloads.
- Give each eval run its own identity: short lived, narrowly scoped and traceable, so anomalous use is attributable and revocable. See the AI Agent Sandboxing and Code Execution Guide.
- Monitor the harness live: alert on outbound connections, account sign-ups, Tor use and messages to real addresses during runs, and have a stop procedure.
- Protect eval data: datasets contain working attack strings and transcripts may contain seeded sensitive data. Restrict access and keep them out of training data unless deliberately included.
Post-release evaluation and monitoring
Production behaviour drifts as providers update hosted models, retrieval corpora change and users find new phrasings. Post-release evaluation has two parts.
- Scheduled suites against a production mirror: same model endpoint, prompts, tools and guardrails, with mocks in place of real downstream systems.
- Online signals from production: guardrail block rates, tool call denials, canary token hits and user reports, sampled into a review queue. Confirmed incidents become new eval cases.
Record provider model identifiers in every production log line so a regression can be tied to a model change. For production monitoring design see the AI Agent Behaviour Baselining and Anomaly Detection Guide.
Implementation roadmap
- Weeks 1 to 4, baseline: pick one high value agent, write its security claims, stand up an open-source harness such as Inspect, and run an initial catalogue of a few hundred cases. Record results only; do not gate yet.
- Weeks 5 to 8, sandbox and identities: move the harness into an isolated environment with no egress, mock services, canary records and fake credentials. Issue a dedicated, short lived identity per run and confirm no production secret is reachable.
- Weeks 9 to 12, report-only gating: wire the smoke suite into CI and publish results on every change, but let builds pass. Use this period to tune thresholds, fix flaky scorers and measure grader agreement.
- Quarter 2, enforce: block merges and releases that fail critical tasks. Allow a documented override by the risk owner with an expiry date, which is your rollback if a gate proves wrong.
- Ongoing, widen and refresh: add agents by risk tier, add adaptive attacks each quarter, re-run on provider model changes and feed every incident into the library.
Where identity fits
Identity shows up in evaluations in two ways. First, as a test target: agents should be tested for whether they read, reuse or leak the credentials they hold, whether they request more access than a task needs, and whether they create accounts or identities. Seed the environment with canary tokens that alert when used, and score any use of them as a failure. Second, as a control on the harness itself: the agent under test runs as a non-human identity, and in the 2026 incidents above it was over-privileged or shared credentials that turned a test into a breach. Scope eval identities to the sandbox, expire them after each run and keep their activity in the same logs your SOC watches. See the Red Teaming AI Agents for Identity Abuse guide for identity focused attack cases.
Framework mapping
| Framework | Where security evaluations fit |
|---|---|
| NIST SP 800-53 Rev. 5 | SA-11 Developer Testing and Evaluation; CA-8 Penetration Testing; RA-5 Vulnerability Monitoring and Scanning; SI-7 for integrity of tested configurations |
| NIST CSF 2.0 | ID.RA-01 (vulnerabilities identified and recorded); ID.IM-02 (improvements from security tests and exercises); DE.CM-09 (monitoring runtime environments) |
| NIST AI RMF and AI 600-1 | The MEASURE function of AI RMF 1.0; the Generative AI Profile (NIST AI 600-1, July 2024) as the companion for generative systems |
| NIST AI 800-2 (draft) | Practices for objectives, running and reporting automated benchmark evaluations |
| AIUC-1 | C002 Conduct pre-deployment testing; B001 Third-party testing for security risk, which the standard describes as at least every three months |
| OWASP | Top 10 for LLM Applications 2025 and Top 10 for Agentic Applications as test categories; GenAI Red Teaming Guide for scope |
| MITRE ATLAS | Technique IDs as case labels, for example AML.T0051 LLM Prompt Injection and AML.T0085 Data from AI Services |
| ISO/IEC 27001:2022 | Annex A 8.29 Security testing in development and acceptance; 8.25 Secure development life cycle |
Common failure patterns
- Testing the model, shipping the agent: passing a jailbreak benchmark says little about whether the agent will call a destructive tool when a retrieved document tells it to. Evaluate the full system with its tools and permissions.
- One aggregate score: a single "92% safe" figure hides the one task where exfiltration succeeds. CAISI found individual task success rates varied widely around a 57% average.
- Tuning on the test set: teams adjust prompts until the suite passes, then a paraphrase succeeds in production. Keep held-out cases and refresh attacks.
- Grader model trusted blindly: an LLM judge that misses subtle leaks produces false confidence. Use state-based checks where possible and audit grader decisions.
- Live access in the harness: real credentials and open egress in test environments, the root cause in the 2026 evaluation incidents above.
- No re-run after provider updates: hosted models change underneath the application; without scheduled runs a regression goes unnoticed until it is exploited.
Practitioner checklist
- Write the security claims each AI system must support and map each to eval categories.
- Build a case library from public benchmarks, red team findings and incidents, labelled with OWASP and ATLAS IDs.
- Test agents in environments that mirror production tools and permissions, with mocks, canary data and fake credentials.
- Score on environment state where possible; audit any grader model.
- Run each attack multiple times and report success per task, success within k attempts, utility and false refusals.
- Gate merges and releases on critical tasks; schedule post-release runs and re-run on every provider model change.
- Deny eval sandboxes network egress and instance metadata by default, and give each run a scoped, short lived identity.
- Turn every confirmed red team or production finding into a regression case before closing it.
Standards and references
- NIST CAISI: Technical Blog: Strengthening AI Agent Hijacking Evaluations
- NIST CAISI: Towards Best Practices for Automated Benchmark Evaluations (NIST AI 800-2 draft)
- NIST: AI 600-1 Generative Artificial Intelligence Profile
- UK AI Security Institute: Inspect
- OWASP GenAI Security Project: GenAI Red Teaming Guide
- OWASP: Top 10 for LLM Applications 2025
- AIUC-1: Security domain requirements
- MITRE ATLAS: LLM Prompt Injection
Related NHI Mgmt Group resources: AI Red Teaming Guide · AI Agent Security Testing Checklist · Prompt Injection Guide · Red Teaming AI Agents for Identity Abuse · AIUC-1 Agent Certification Guide