LLM selection is the process of choosing which large language model to use for a specific task, based on security, accuracy, latency, cost, and control requirements. In identity and AI security, selection should align model capability with data sensitivity, tool access, governance controls, and risk tolerance across agentic workflows.
How LLM selection shapes security and governance
LLM selection is not just a model-performance choice, it is a control decision. The model you pick determines how much sensitive context enters the prompt, what tools the system can invoke, how outputs are handled, and how much organisational risk is acceptable for the task.
That is why model choice should be tied to the job the system must do. A low-risk summarisation task, a customer-facing assistant, and an agent with write access to internal systems do not require the same model profile, even if they all use the same application stack.
Capability, cost, and latency trade-offs
Selection usually balances three operational constraints, accuracy, latency, and cost. Stronger models can improve reasoning or instruction-following, but they may also increase spend, response time, or dependency on a particular provider. Weaker models can be appropriate when the task is narrow, bounded, and tolerant of occasional error.
The important point is that capability should be matched to task complexity, not assumed globally. If a workflow needs consistent classification, extraction, or policy-aware responses, the safer choice is often the model that is most predictable under those constraints, not the one with the highest general benchmark score.
Data sensitivity and tool access requirements
LLM selection becomes more consequential when prompts contain confidential, regulated, or privileged information. If a model will see customer records, internal documents, code, or secrets-adjacent material, the decision must consider retention, isolation, tenant boundaries, and whether the provider can meet the organisation’s data handling expectations.
Tool access changes the equation again. Once a model can call APIs, retrieve records, or trigger actions, selection is no longer only about language quality. It also becomes a question of authorization scope, blast radius, and whether the model is suitable for the specific permissions and guardrails attached to that workflow.
For governance-driven model evaluation, NIST AI RMF provides a useful external reference point, and the NIST AI 600-1 GenAI Profile is especially relevant when LLM choice must align with testing, disclosure, and operational risk management.
Selection criteria for agentic workflows
In agentic systems, the model is part of an execution chain, not just a text generator. Selection should account for whether the model can reliably follow policies, resist prompt manipulation, work with constrained context, and operate safely when paired with memory, tools, and external data sources.
That is why some environments standardise on different models for different stages, for example one model for drafting, another for classification, and a more tightly governed model for action-bearing steps. The goal is to reduce unnecessary privilege and keep the highest-risk capabilities inside the smallest possible trust boundary.
The OWASP Agentic AI Top 10 is a useful companion when model choice affects identity, privilege, tool misuse, or other agentic failure modes, and the NIST AI Risk Management Framework helps anchor the broader governance decision.
Risk and Threat Considerations
LLM selection can create exposure when teams choose a model for convenience rather than control. A model that is too permissive, poorly governed, or exposed to sensitive context can increase data leakage, tool abuse, and the impact of prompt injection or credential exposure in downstream workflows.
Failure mechanism: The model is placed inside a workflow whose data sensitivity or action scope exceeds the model’s isolation, policy, or safety assumptions, so unsafe prompts, leaked context, or excessive tool reach can be exploited.
Impact: Organisations can lose confidentiality, trigger unauthorised actions, or expand the blast radius of an agentic compromise, especially where the chosen model is tightly coupled to privileged systems or shared conversation state.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | LLM selection is an AI risk governance decision |
| Recommendation — Apply AI RMF governance to tie model choice to risk, testing, and accountability. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Model choice affects agent authority and tool access risk |
| ASI02 — Tool Misuse | The selected model may misuse tools if its action scope is too broad | |
| Recommendation — Limit model-driven actions to the smallest privilege set needed for the workflow. Constrain tool exposure and validate tool calls before execution. | ||
| NIST SP 800-53 Rev 5 | IA-5 — Authenticator Management | Model-connected workflows often depend on secrets and credential handling |
| AC-6 — Least Privilege | Selection should match the minimum authority needed for the LLM task | |
| Recommendation — Manage and rotate credentials used by model-integrated services and agents. Enforce least privilege for any model, connector, or agent that can act on data. | ||
Practitioner Guidance
Why practitioners should care: Treat LLM selection as part of security architecture, not only product procurement. The right choice depends on whether the workflow is read-only, data-sensitive, tool-enabled, or capable of taking action on behalf of a user or service.
Governance implication: Define approval criteria that tie model choice to data class, allowed tools, and operational impact, then review those criteria whenever the workflow changes. A model that is acceptable for one use case may be inappropriate once prompts, connectors, or permissions expand.