How should organisations evaluate foundation models for specific business use cases?

Why This Matters for Security Teams

Foundation model evaluation is not just a model quality exercise. It is a control decision that affects customer trust, data exposure, legal exposure, and operational resilience. A model that performs well in generic testing can still be unsafe in a specific workflow if it mishandles sensitive prompts, produces confident but wrong output, or behaves inconsistently under pressure. Current guidance suggests evaluating the model in the same policy context, user role, and data environment where it will actually operate, rather than relying on broad public benchmarks alone. The NIST AI 600-1 Generative AI Profile is useful here because it frames GenAI risk around governance, measurement, and monitoring rather than one-time approval.

Security teams often miss the difference between model capability and business suitability. A model may answer well in a lab but still violate internal policy when it is connected to real users, retrieval sources, or downstream automation. That creates a false sense of confidence if the evaluation does not include the actual prompts, the real escalation paths, and the business owner’s acceptance criteria. In practice, many security teams encounter model failure only after the first production misuse report, rather than through intentional use-case validation.

How It Works in Practice

Effective evaluation starts by defining the use case in operational terms: who will use the model, what data it can see, what decisions it may influence, and what failure would cost the business. That means testing for more than accuracy. Teams should assess refusal behaviour, prompt sensitivity, hallucination impact, privacy leakage, policy bypass, and the model’s tendency to produce unsafe or unapproved content in the exact workflow it will support. For high-risk use cases, evaluation should also check how the model behaves when users try to override controls or extract restricted information.

A practical evaluation plan usually includes:

• Scenario-based tests built from real workflows, not synthetic prompts only.
• Safety and privacy checks aligned to the data the model will actually process.
• Adversarial testing for prompt injection, jailbreak attempts, and instruction conflicts.
• Review of output quality by both technical owners and business stakeholders.
• Approval criteria that define what is acceptable, what requires mitigation, and what blocks deployment.

Security control mapping should be explicit. The NIST SP 800-53 Rev 5 Security and Privacy Controls helps translate model evaluation into familiar control language around access, auditing, integrity, and privacy. For example, if a model will summarize internal documents, the evaluation should verify that it does not expose restricted content, does not retain sensitive inputs beyond policy, and does not generate outputs that break approval rules. Where the model is wrapped in RAG or workflow automation, the evaluation must include the retrieval layer and the downstream action layer, because model quality alone does not prove system safety.

Best practice is evolving toward continuous validation, not one-off signoff. Models should be re-tested when prompts, data sources, guardrails, or user populations change, and when the vendor updates the underlying model. These controls tend to break down when a model is promoted from pilot to production without re-running business-specific tests on live integrations and real permission boundaries.

Common Variations and Edge Cases

Tighter evaluation often increases delivery time and review overhead, requiring organisations to balance deployment speed against business risk. That tradeoff becomes sharper when the model is externally hosted, frequently updated, or used across multiple business units with different risk tolerances. There is no universal standard for use-case approval thresholds yet, so organisations should treat benchmark scores as inputs, not as a decision on their own.

Some use cases need stricter treatment than others. A customer-facing assistant, an internal analyst copilot, and a workflow agent that can trigger actions all require different evidence. For agentic or tool-using systems, the evaluation should extend beyond text quality to execution authority, escalation handling, and whether the model can be constrained to its intended role. Where the model touches identity, access, or privileged workflows, the review should also ask whether the model can surface secrets, infer sensitive attributes, or influence an approval path it should not control.

Organisations should also distinguish between model risk and integration risk. A safe model can still become unsafe if prompts are poorly designed, retrieval sources are unvetted, or logs store sensitive content without protection. In those cases, the right answer is not necessarily to reject the model, but to narrow the use case, add compensating controls, or require human review before action. NIST AI 600-1 Generative AI Profile remains most useful when paired with a clear internal policy for who can approve exceptions and how revalidation occurs after any material change.

Related resources from NHI Mgmt Group

• How should organisations measure trust across AI use cases, agents, and models?
• How should organisations evaluate open-source platforms for identity and security use cases?
• Should organisations use new AI-specific identity standards or existing ones?
• Should organisations use business impact to prioritise identity risk?