How should teams build an offline eval for an AI system that feels unreliable?

Why This Matters for Security Teams

An AI system that feels unreliable is often already creating operational risk, even if no formal incident has been declared. Offline evaluation turns that discomfort into evidence. It helps teams separate genuine model weakness from bad prompts, weak retrieval, broken routing, data drift, or inconsistent human review. For security and governance teams, that distinction matters because unreliable outputs can affect access decisions, customer-facing actions, fraud decisions, and incident workflows. The control mindset aligns well with NIST SP 800-53 Rev 5 Security and Privacy Controls, especially where organisations need repeatable testing, accountability, and change management.

The main mistake is treating evals as a one-time benchmark instead of a living assurance process. Teams often build a large but uncurated test set, then trust a single score that hides where the system fails. That gives false confidence and makes regressions hard to explain. A better approach is to define the target behaviour, the failure modes that matter, and the acceptable tradeoffs before writing the first test case. In practice, many security teams encounter AI unreliability only after production users have already worked around it, rather than through intentional offline validation.

How It Works in Practice

A useful offline eval starts with a narrow question such as, “Does this system answer policy questions consistently enough to support first-line triage?” That question determines the dataset, the scorers, and the acceptance threshold. For an AI system, the dataset should be small at first, but representative of the real operating context: normal requests, edge cases, ambiguous prompts, adversarial inputs, and examples that reflect known failure patterns such as hallucination, refusal errors, or unsafe tool use. The goal is to capture the shape of the risk, not to maximise volume.

Scorers should map to the dimensions that matter for the use case. Common dimensions include factuality, completeness, policy compliance, citation quality, tone, and consistency. For retrieval-augmented systems, teams often also measure whether the model used the right source material, because good retrieval can still produce bad answers if reasoning is weak. Where the system touches security-sensitive workflows, the eval should also check for prompt injection resilience, disclosure of secrets, and unsafe instruction following. For AI governance, that is consistent with current guidance in the NIST AI Risk Management Framework and the OWASP Top 10 for Large Language Model Applications.

• Define the system boundary before scoring the output.
• Create a versioned test set that reflects real use, not just clean examples.
• Use multiple scorers so one weak dimension does not mask another.
• Record prompt, model, retrieval context, and tool state for every run.
• Compare changes across versions, not against intuition.

Best practice is evolving around whether to use human-only scoring, model-assisted scoring, or hybrid approaches. The right answer depends on risk, cost, and reproducibility needs. In regulated environments, offline evals should be paired with change approval and audit evidence, so the results are traceable across model, prompt, and data versions. These controls tend to break down when the system depends on live external tools or rapidly changing knowledge sources because the offline dataset cannot fully reproduce runtime conditions.

Common Variations and Edge Cases

Tighter evaluation coverage often increases review time and annotation cost, requiring organisations to balance confidence against delivery speed. That tradeoff is especially sharp for agentic AI, where the system may plan, call tools, and modify state rather than only generating text. In those cases, a simple answer-quality score is not enough. The eval must also assess whether the agent chose the right action, respected boundaries, and recovered safely from ambiguous instructions. For agentic workflows, security teams should also consider identity and privilege design for the agent itself, because execution authority can become a hidden failure point.

There is no universal standard for how many samples are “enough.” Current guidance suggests using enough coverage to detect the failures that matter most, then expanding as the system matures. Some teams over-focus on aggregate accuracy and miss critical tail risks. Others optimise for adversarial robustness and ignore ordinary user confusion. The best offline evals reflect the actual operating envelope and are reviewed alongside incident data, user feedback, and release notes. For systems with high regulatory exposure, link the eval criteria to documented assurance duties under AI governance and operational resilience expectations, rather than treating the benchmark as a standalone artefact.

For broader AI security practice, NIST and OWASP guidance should be read as complements, not substitutes, because one set focuses on risk management and the other on concrete failure patterns. Where the model is embedded into a product subject to formal assurance or lifecycle controls, teams should preserve the full eval history and treat regressions as release blockers until the cause is understood.

Related resources from NHI Mgmt Group

• How should security teams limit AI system damage when model refusals are unreliable?
• Why do AI eval criteria change after teams see the system in production?
• Should teams build their own permissions system or use an authorization service?
• How should security teams build an AI asset inventory for governance?