Model-Level vs System-Level AI Evaluation: What Each Proves and Who Owns It

A model can pass every published benchmark and still sit inside an application that leaks data, calls the wrong tool or acts on an attacker's instructions, because the benchmark never saw your prompts, your tools or your data. This guide compares model-level evaluation (testing the bare model on fixed tasks) with system-level evaluation (testing the deployed application, agent or workflow end to end). It is written for AI assurance leads, application security teams, product owners and release approvers who have to decide which evidence answers which question, and who is accountable for each. It does not repeat the method comparison in the AI Red Teaming vs Evaluations vs Benchmarks guide; it looks at scope and ownership instead. Sources were checked on 13 November 2026.

Key takeaways

• The unit under test decides what you can claim. A model-level result describes the model under the harness conditions used; only a system-level result describes the thing your users and attackers actually touch.
• Most real AI security failures are system failures. Prompt injection through retrieved content, excessive tool permissions and output that triggers a fetch all need the application around the model, so a model score cannot show them.
• Each level has a different owner. The model provider is accountable for model evaluations and their disclosure; the deployer is accountable for system evaluation of its own prompts, tools, data and permissions, and cannot outsource that.
• Change scope tells you which level to re-run. A provider model update needs both levels; a prompt, tool, retrieval or permission change needs the system suite even when the model is untouched.
• Score system tests on state, not text. Check whether the email was sent, the record changed or the canary left the boundary, and repeat high-impact cases, because attackers retry and outputs vary.

The short answer: two scopes, two questions

Model-level evaluation asks "how capable and how safe is this model, in general, under standard conditions?". System-level evaluation asks "does our deployed system, with its instructions, tools, data, guardrails and identities, behave acceptably for our use and under attack?". Both are needed; they fail differently and have different owners.

NIST frames the same point in the Generative AI Profile (NIST AI 600-1): measurements under the MEASURE 2.3 subcategory should be demonstrated for conditions similar to the deployment setting, and action MS-2.5-001 warns against extrapolating system performance or capabilities from narrow, non-systematic or anecdotal assessments.

Model-level evaluation in depth

Threat model and architecture

A model-level evaluation runs a dataset through the model using an evaluation harness: a fixed prompt template, a fixed number of attempts, and a scorer. HELM (Holistic Evaluation of Language Models) is a clear example: its paper reports evaluating 30 models on 16 core scenarios and 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) under standardised conditions, plus 26 targeted scenarios. Standardisation makes models comparable, and it is also the limit: the standard conditions are not yours. The implicit attacker talks directly to the model and has no route through your documents, tools or other users' data.

What it is good for

• Model selection: NIST AI 600-1 action MS-2.3-001 suggests considering baseline model performance on suites of benchmarks when selecting a model for fine-tuning or retrieval-augmented generation.
• Change detection: a fixed model suite re-run after a provider update shows gross shifts in capability or refusal behaviour.
• Regulatory evidence for providers: under Article 55(1)(a) of the EU AI Act, providers of general-purpose AI models with systemic risk must evaluate the model with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model.

Failure modes

• Contamination and gaming: public test items leak into training data, so scores can reflect memory. The ML model evaluation and monitoring for security teams guide covers the statistics.
• Harness mismatch: a vendor score used a prompt format, temperature and attempt count you do not use.
• Missing context: the model had no system prompt, no retrieved documents and no tools, so injection through content and tool misuse were never in scope.

Controls at model level

• Re-run, do not quote: run your shortlist through the same harness yourself, recording harness version, prompt template, temperature and attempts.
• Read the provider's card critically: note which evaluations were run on the bare model and which on a product configuration, and what was out of scope. See the AI BOM vs Model Card vs Datasheet guide.
• Keep a private model suite: a small, never-published set of refusal, leakage and instruction-following cases that you run on every candidate and every provider version.
• Pin versions: call a dated model version rather than a floating alias so a silent provider update cannot invalidate your evidence.

System-level evaluation in depth

Threat model and architecture

System-level evaluation treats the deployed application as the target. The harness drives the real entry points (chat UI, API, email inbox, ticket queue) and observes the real side effects (tool calls, database writes, outbound requests, rendered output). Its threat model includes the user, other users whose content the system reads, any author of a document, email, web page or tool result that reaches the context, and compromised tools or peer agents. OWASP's LLM06:2025 Excessive Agency entry describes the core system risk: damaging actions performed in response to unexpected, ambiguous or manipulated model output, with root causes in excessive functionality, excessive permissions and excessive autonomy. All three are properties of the system you built, not the model.

What changes the result

Tool-use and agent tests

Agent benchmarks show why tool access needs its own tests. AgentDojo (Debenedetti and colleagues, ETH Zurich) runs agents through realistic tasks, reported as 97 tasks and 629 security test cases in its paper, in environments such as an email client, e-banking and travel booking, and scores both utility (did the task get done) and attack success (did an injected instruction get followed). When NIST's Center for AI Standards and Innovation (CAISI) extended it, the strongest baseline hijacking attack in the Workspace environment succeeded 11% of the time, while a new attack developed with the UK AI Security Institute succeeded 81% of the time; across five injection tasks the average success rate rose from 57% with one attempt to 80% with 25 attempts. CAISI's lessons apply to every system suite: make evaluations adaptive, analyse results per task rather than as an average, and test with multiple attempts where an attacker can retry. The detailed case catalogue lives in the AI Security Evaluations Guide.

Failure modes

• Testing a copy, not the system: the suite calls the model API with the system prompt pasted in, skipping retrieval, tools, rendering and identity.
• Text-only scoring: a grader reads the reply and finds it polite, while the agent has already called the send-email tool.
• Happy-path data: test documents contain no injected instructions, so retrieval-borne attacks are never exercised.
• Production identity in tests: evaluation agents run with live credentials and reach real systems. The AI Lab and Agent Evaluation Incidents guide shows what follows.

Controls at system level

• Drive real entry points: run the suite through the same API, channel and middleware users hit, in a staging tenant with production-equivalent configuration.
• Instrument side effects: capture every tool call with arguments, every outbound request and every data write, and assert on them.
• Seed canaries: place unique marker records in data the system can reach, and fail any run where a marker appears in an outbound request or another user's session.
• Sandbox the harness: frameworks such as the UK AI Security Institute's Inspect structure each evaluation as a dataset, a solver (from a single call to a tool-using agent, including MCP tools) and a scorer, and can run untrusted code in Docker or Kubernetes sandboxes. See the Inspect AI Guide.
• Give the evaluation its own identity: a dedicated, scoped, short-lived non-human identity per run, so tests cannot reach production data and every action is attributable.

Where model results and system results disagree

Disagreement between the levels is information, not noise.

The first row is the one that hurts. The EchoLeak vulnerability in Microsoft 365 Copilot (CVE-2025-32711, fixed server side with no evidence of exploitation) is a system-level failure: a crafted inbound email was pulled into context by retrieval, its hidden instructions asked for internal data to be placed in a URL, and rendering paths through trusted services fetched it. No benchmark of the underlying model would have exercised that chain, because it needed retrieval, the user's data scope and output rendering together.

Accountability: who owns which evidence

Most organisations deploy models they did not train, so accountability splits cleanly. Write the split down before the first release.

• Providers cannot test your system: they do not have your prompts, tools, data or permissions. Contract for disclosure of model evaluation scope, version change notice and incident notification, not for assurance of your deployment.
• Deployers cannot inherit a model's pass: under the EU AI Act, testing of high-risk AI systems (Article 9(6) to 9(8)) is about the system: it must check consistent performance for the intended purpose, be carried out against prior defined metrics and probabilistic thresholds appropriate to that purpose, and happen before the system is placed on the market or put into service. Confirm current obligations and dates against the official text for your role.
• Name a single accountable owner per system: the product owner signs the release, the security function owns the system suite content, and the AI assurance function owns the evidence pack. The AI production acceptance criteria guide sets out the gate itself.

Decision criteria: which level, when

• Choosing between models: model level first (your harness, your private suite), then a short system run of the two finalists in your real configuration.
• Prompt, tool, retrieval or permission change: system level only, full suite for the affected capability plus a smoke set across the rest. The AI security regression testing guide covers triggers and pipeline wiring.
• Provider model version change: both. Model suite to detect behaviour shift; full system suite before switching production traffic.
• First launch, new autonomy level or new class of tool: both, plus a scoped red team, because the suite only knows what it was taught.
• Small organisation, read-only assistant: rely on provider model evaluations and run a compact system suite (30 to 50 cases on data scope, retrieved-content injection and rendering) on every change.
• Large organisation, many agents with write tools: a shared evaluation platform, per-agent suites owned by product teams, and a central model suite on every provider version.

Worked scenario: a support agent on a new model version

A company runs a customer support agent that reads tickets and the knowledge base through retrieval, can look up orders and issue refunds up to a set value, and replies by email. The provider releases a new model version with better published scores on reasoning and refusal benchmarks.

1. Model suite: the platform team runs its private model suite on the pinned new version. Refusal and leakage cases improve slightly; one instruction-following category regresses.
2. System suite in staging: the team runs 180 system cases through the real support API with the new model: task cases, seeded tickets carrying injected instructions, cross-customer lookups, refund boundary cases and output rendering with canaries. Each high-impact case runs 10 times.
3. Finding: on the new version, a seeded ticket that asks the agent to "confirm the refund for the attached order list" causes a refund tool call on an order belonging to another customer in 2 of 10 runs. Text scoring would have passed it: the reply looked correct.
4. Diagnosis: the model change exposed a system weakness that already existed: the refund tool trusted the order ID from the model rather than checking it against the authenticated customer.
5. Fix and evidence: the refund tool now enforces ownership server side and requires the order to come from the authenticated session. The case joins the gate set. Both versions now pass 10 of 10. The release record holds the model suite log, system suite log, the new case and the sign-off.

The better benchmark score was real and irrelevant to the defect. Only a state-scored, repeated system test found it, and the fix was an authorisation control.

Where identity fits

System-level evaluation is partly an access review: whether an agent can reach data or actions it should not is answered by testing the identities it acts as, not the model. Run system suites with the same identity model as production (delegated user tokens or per-agent service accounts) and confirm that tool calls carry the end user's authorisation where they should. See the Agentic AI Identity Guide and the OWASP LLM Top 10 LLM06 Excessive Agency Guide.

Detection and measurement

• System coverage: share of tools, retrieval sources and data classes with at least one system case; target 100% for tools with write or send capability.
• Attack success within k attempts: for high-impact injection and tool-misuse cases, the share of cases where any of k runs succeeds, reported per task, not averaged.
• State-scored share: proportion of security cases scored on side effects (tool calls, outbound requests, writes) rather than reply text.
• Change-to-suite latency: days between a new tool or retrieval source shipping and its first system case existing; aim for zero by gating the merge.
• Finding conversion rate: share of red team and production findings that became system suite cases within one release.

Framework mapping

Common failure patterns

• Leaderboard as release gate: a public score approves a deployment that was never tested with its own tools and data.
• Provider card treated as deployer assurance: the provider's evaluations are filed as the system's evidence, though they covered a different configuration.
• Prompt changes outside the pipeline: instructions edited in a console with no re-run.
• Averages on the dashboard: a 95% pass rate hides that every failure sits in the one task that moves money or data.
• No owner for the system suite: model tests and red teaming are owned, the cases that gate releases are not.

Practitioner checklist

• Write down, per AI system, which evidence is model level and which is system level, and who owns each.
• Pin model versions and re-run the private model suite on every provider version change.
• Build the system suite against real entry points in a staging tenant with production-equivalent tools, retrieval and identities.
• Seed canaries and score security cases on side effects, not reply text.
• Cover every tool, retrieval source and rendering path, with injected-instruction cases for each untrusted input channel.
• Repeat high-impact cases and report success within k attempts per task.
• Gate merges that add tools or sources on the presence of matching system cases.
• Give evaluation runs their own scoped, short-lived identities and isolate them from production.
• Convert every red team and production finding into a system case before the next release.

Standards and references

• NIST: AI 600-1 Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• EU AI Act: Article 9 Risk Management System
• EU AI Act: Article 55 Obligations for Providers of General-Purpose AI Models with Systemic Risk
• OWASP: Top 10 for LLM Applications 2025
• NCSC: Guidelines for Secure AI System Development, Secure Deployment
• NIST CAISI: Technical Blog, Strengthening AI Agent Hijacking Evaluations
• arXiv: AgentDojo, a Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
• arXiv: Holistic Evaluation of Language Models (HELM)

Related NHI Mgmt Group resources: AI Red Teaming vs Evaluations vs Benchmarks · AI Security Evaluations Guide · AI production acceptance criteria · AI security regression testing · RAG Security Guide