How should teams decide which AI model to use for a workflow?

Why This Matters for Security Teams

Model selection is an operational control, not just a procurement choice. A workflow that looks inexpensive on paper can become costly when it produces more retries, longer latency, extra human review, or inconsistent outputs that trigger downstream exceptions. Security teams should evaluate the model against task reliability, data sensitivity, and governance constraints, then measure performance on the exact workload rather than assuming a general benchmark will hold.

This is especially important where AI output affects access decisions, customer-facing actions, or security operations. In those cases, poor model fit can create false confidence, noisy alerts, or policy drift that is hard to detect after deployment. A practical evaluation also needs ownership: who approves the model, who monitors drift, and what happens when quality drops. The governance lens in NIST Cybersecurity Framework 2.0 is useful here because it forces teams to connect technology choices to measurable risk outcomes.

In practice, many teams discover the real failure mode only after the workflow is already producing costly exceptions, rather than through a deliberate model selection process.

How It Works in Practice

The most defensible method is to treat model choice as a workload test, not a static ranking exercise. Start by defining the task clearly: classification, summarisation, extraction, code generation, agentic tool use, or a mixed workflow. Then test a small candidate set against representative inputs, including edge cases, adversarial prompts, and low-quality data. Measure the full end-to-end result, not just the first response.

Useful criteria usually include task success rate, retry rate, latency, human escalation rate, output validation effort, and any security or privacy constraints tied to the data. For AI systems that interact with tools or downstream systems, teams should also assess whether the model is reliable under tool failures, ambiguous instructions, and changing context. NIST AI Risk Management Framework is a strong reference point for connecting model behaviour to governance and accountability, while the OWASP Top 10 for Large Language Model Applications helps teams think about failure modes such as prompt injection, insecure output handling, and excessive agency.

• Choose a representative workload sample, not a curated demo set.
• Score success against business outcomes, not token counts alone.
• Track retries, timeouts, refusals, hallucinations, and human corrections.
• Check whether the model can be safely constrained for the required data class.
• Re-test when prompts, tools, or policies change.

When routing multiple models, many teams use a small, explicit policy such as simple tasks to a cheaper model, complex or sensitive tasks to a stronger model, and high-risk actions to a model plus human approval. For agentic workflows, model choice also intersects with identity and privilege because the model may inherit tool access or invoke actions on behalf of a user or system. Guidance from CISA secure AI development resources is helpful for grounding those control decisions in operational safeguards. These controls tend to break down when the workflow depends on rapidly changing prompts, unstructured external data, or hidden downstream automation because the true cost of failure is no longer visible at the model layer.

Common Variations and Edge Cases

Tighter model governance often increases evaluation overhead, requiring organisations to balance faster deployment against stronger assurance. Best practice is evolving for dynamic routing, especially where teams mix frontier models, smaller specialised models, and retrieval-augmented generation in the same workflow. There is no universal standard for this yet, so the right answer depends on risk appetite, data sensitivity, and how much variance the business can tolerate.

Some edge cases deserve extra scrutiny. For regulated or high-impact workflows, a slightly more expensive model may still be the right choice if it reduces exception handling or improves auditability. For agentic systems, the model may appear strong in isolated tests but fail once tool permissions, memory, or external APIs are introduced. For retrieval-heavy workflows, the bottleneck may be the retrieval layer rather than the model itself, so changing models alone may not improve outcomes. In those cases, alignment to NIST Cybersecurity Framework 2.0 helps teams keep the decision tied to resilience, monitoring, and recovery rather than raw model capability.

The practical rule is simple: use the model that delivers acceptable quality at the lowest measured end-to-end cost for that specific workflow, then revisit the decision as data, threats, and business requirements change.

Related resources from NHI Mgmt Group

• How should security teams govern AI agents that use Model Context Protocol?
• How do IAM teams decide whether an AI use case needs new controls or better NHI hygiene?
• How should security teams govern AI use when the same model creates different risk in different contexts?
• Should healthcare teams use the same zero trust model for AI agents and service accounts?