A model router decides at request time which model, and often which provider and region, handles each prompt, so the security properties you tested on one model are no longer guaranteed for the response your user receives. Routing promises lower cost, better availability and the right model for each task. It also turns model selection into a runtime decision that an attacker can influence, that a fallback can widen, and that your guardrails, evaluations and data agreements must cover for every model in the pool. This guide compares routed (dynamic) model selection with single-model (pinned) deployments for AI platform owners, security architects and data protection leads. It goes beyond the credential model in the Model Aggregator and LLM Router Key Security guide and the routing basics in the AI Gateway and LLM Gateway Guide, and focuses on the selection decision itself. Product examples are listed alphabetically; order is not a ranking. Sources were checked on 13 November 2026.
Key takeaways
- Every model in the pool is in scope. A routed deployment is only as safe as the weakest model it can reach, including fallbacks, so evaluation, guardrails and data terms must cover all of them.
- The router is a control plane an attacker can steer. Research on rerouting LLM routers showed query-independent token sequences that push a router towards a chosen model in white-box and black-box settings, and perplexity filtering did not stop them.
- Fallback is where policy leaks. A fallback on error, context length or content-policy refusal can send a prompt to a different provider, region or safety profile than the primary; constrain fallbacks to an approved set.
- Log the model that answered, not the model you asked for. Without the selected model, provider and region on every record, you cannot investigate an incident, prove residency or reproduce behaviour.
- Pin where consistency is the requirement. Regulated decisions, agents with write tools and workloads under strict residency usually belong on a single pinned model, with routing kept for low-risk, high-volume traffic.
The short answer: what changes when selection becomes dynamic
| Dimension | Single-model deployment | Routed deployment |
|---|---|---|
| What the caller gets | One named, pinned model version at one provider endpoint | A logical endpoint; the router picks the model, and sometimes the provider and region, per request |
| Evaluation scope | One model plus your system | Every model in the pool, every fallback path, and the routing policy itself |
| Guardrail consistency | One behaviour profile to tune against | Refusal, instruction-following and leakage behaviour vary by model; external guardrails must be model-agnostic |
| Data path | Known provider, region and retention terms | Depends on the route taken; must be constrained so every route meets the strictest requirement |
| Cost | Predictable per token | Lower on average, but open to cost inflation if routing can be steered towards expensive models |
| Availability | Provider outage is your outage unless you add failover | Failover built in, at the price of policy drift during failover |
| Reproducibility | Same model for the same request | The same prompt may go to different models on different days |
| New attack surface | None beyond the model endpoint | The routing classifier, its configuration, fallback rules and the router's own keys and logs |
How routers decide, and what that means for security
Routers fall into three broad designs, often combined in one product:
- Learned quality routers: a classifier predicts which model will answer a prompt well enough at the lowest cost. RouteLLM (Ong and colleagues) trains routers on human preference data to choose between a stronger and a weaker model and reports cost reductions of over two times in some cases without loss of response quality. The security consequence: a statistical model, which can be manipulated, now decides which model processes your data.
- Rule and metric routers: fixed rules choose by price, latency, throughput, error rate or a weighted shuffle across deployments. They are easier to reason about but still change the destination when load or price changes.
- Fallback chains: an ordered list of alternatives tried when the primary fails, times out, exceeds its context window or refuses on content policy grounds.
Examples of routing mechanisms
The table lists each product's own routing and restriction settings from its documentation, alphabetically. For hardening of each product see the linked independent guides.
| Product feature | How it selects | Settings that constrain the pool |
|---|---|---|
| Amazon Bedrock intelligent prompt routing | Predicts response quality for models in the same family and routes away from the configured fallback model only when another model's predicted quality exceeds it by more than the response quality difference you set | Configured routers with chosen models and fallback model; single-region or cross-region inference profiles (for example the us. prefix) that keep requests within a geographic area |
| Azure model router (Microsoft Foundry) | A trained model analyses each prompt and picks an underlying model; routing modes Balanced (default), Cost and Quality | Model subset (also the fallback set, so prompts are not processed by unapproved models); Global Standard or Data Zone Standard deployment types, with routing honouring data zone boundaries; Azure Policy model deployment policy applies to the subset |
| LiteLLM Proxy | Configured routing strategies across deployments plus ordered fallbacks: fallbacks, context_window_fallbacks, content_policy_fallbacks and default_fallbacks | Per-key, team and project models lists and access groups; the proxy checks each fallback target against the caller's access (enforce_fallback_model_access) and budget; disable_fallbacks per request |
| OpenRouter provider routing | By default weighs price, throughput, latency, reliability and tool-call quality per provider; sort by price, throughput or latency on request | order, only, ignore, allow_fallbacks (default true), data_collection deny, zdr true, require_parameters, quantizations, max_price; account-wide privacy settings that per-request zdr cannot override |
Three details from these documents matter for security design. Azure's model router documentation notes that an auto-updating router version receives new models over time, which can change the underlying model set, and that new base models are not added to a custom subset unless you include them. OpenRouter's documentation states that its data policy tags on model pages are not a definitive source of third-party policies. LiteLLM's content_policy_fallbacks exist specifically to retry a request that one provider rejected for content policy on another.
See the Amazon Bedrock Security Guide, Azure OpenAI and Azure AI Foundry Security Guide, LiteLLM Proxy Security Guide and, for OpenRouter keys, Model Aggregator and LLM Router Key Security.
Threat model: routed deployments
| Threat | How it happens | Impact | Primary control |
|---|---|---|---|
| Downgrade to a weaker model | Prompt content steers a quality router towards a cheaper model with weaker refusal or instruction-hierarchy behaviour, or load and price shift traffic there | Jailbreaks (ATLAS AML.T0054) and injections that the primary would resist succeed | Only admit models that pass the same system suite; force a pinned model for sensitive routes |
| Upgrade for cost inflation | Confounder gadgets appended to prompts push the router to the expensive model; Shafran and colleagues showed this works in white-box and black-box settings without degrading answers | Denial of wallet (OWASP LLM10:2025, ATLAS AML.T0034 Cost Harvesting) | Per-key budgets, per-route model caps, alert on shifts in model mix |
| Data to an unapproved provider or region | Fallback or default routing reaches a provider without a data agreement, with training on prompts allowed, or outside the required region | Unlawful transfer, contractual breach, loss of confidentiality | Provider and region allowlists enforced at the router, fallbacks restricted to the same set, fail closed |
| Guardrail bypass by refusal fallback | A content-policy refusal triggers a retry on another model with a different safety profile | Harmful or out-of-policy output that one model refused | Disable content-policy fallback for user-facing routes, or route refusals to a model with equal or stricter policy |
| Silent pool change | An auto-updating router or alias adds or swaps models without a change ticket | Untested model in production; evaluation evidence no longer valid | Frozen router versions or explicit inclusion lists; treat pool changes as configuration changes |
| Router compromise or misconfiguration | Router admin keys, configuration store or routing rules altered | All traffic redirected, logs disabled, limits removed | Treat the router as privileged infrastructure: admin keys vaulted, configuration as code, change alerts |
| Context and capability mismatch | Requests route to a model with a smaller context window, no tool-calling support or different tool-call formats | Truncated instructions, dropped system prompt content, malformed tool calls | Require parameter and capability support on every route; test long-context and tool cases per model |
Threat model: single-model deployments
Pinning removes routing risk but keeps or concentrates others.
- Single provider dependency: an outage, quota limit or regional incident stops the service. Teams under pressure then add an unreviewed emergency fallback, which is the routed risk without the controls.
- Model retirement: pinned versions are retired by providers on their schedule, forcing a migration that needs the same evaluation as a router pool change.
- Floating aliases: a deployment that calls an alias such as "latest" is not truly single-model; the provider can change the model behind it.
- Over-provisioned model: sending every request to the most capable model increases cost and the impact of a leaked key, since every call spends at the highest rate.
- Concentrated blast radius: a jailbreak that works on the one model works on all traffic, with no diversity to limit it.
Failure modes compared
| Failure | Single model | Routed |
|---|---|---|
| Untested model answers users | Only if the alias floats or a provider swaps the version | Whenever the pool, fallback set or router version changes without re-evaluation |
| Data leaves the approved boundary | Only through misconfiguration of the one endpoint | Through any route or fallback that is not constrained |
| Inconsistent refusals | Rare; the same model behaves broadly the same way | Common; users and attackers can find the most permissive route by retrying |
| Cost spike | Volume-driven | Volume-driven or routing-driven (cost inflation) |
| Outage | Full, unless failover exists | Partial; failover may breach policy |
| Forensics | Straightforward | Impossible without per-request selected-model logging |
Controls for routed deployments
Constrain the pool
- Approved model register: every model, version, provider and region that a route can reach is listed, owned and evaluated. Anything not on the register is unreachable by configuration, not by convention.
- Allowlisted routes: set the pool explicitly (a model subset, an only list, per-key models lists) rather than relying on ignore lists, which admit every new model by default.
- Fallbacks inside the boundary: fallback targets come from the same approved set and meet the same residency, retention and contractual terms as the primary. Where no compliant fallback exists, fail closed.
- Data terms at request and account level: where the router supports it, set retention and data-use restrictions account-wide and per request, and verify against each provider's own terms rather than the router's summary.
Make policy model-agnostic
- Guardrails outside the model: input and output checks, tool authorisation and data scoping must sit in front of and behind the router, so their behaviour does not depend on which model answered. See the AI Guardrails vs AI Gateways guide.
- Per-model policy where behaviour differs: keep a per-model profile (context window, tool-calling support, known weaknesses from your suite) and route sensitive request classes only to models that meet it.
- Route by data classification: classify the request first (for example, personal data present, regulated data present, tools required) and map each class to a fixed model or a narrow pool.
Evaluate the pool and the router
- Same suite for every model: run your system evaluation suite against each model in the pool individually, then through the router. The Model-Level vs System-Level AI Evaluation guide explains why the system run matters.
- Adversarial routing tests: include cases that try to steer selection (padding, appended token sequences, prompts that mimic complexity) and assert that sensitive classes never reach models outside their pool.
- Refusal-retry tests: confirm that a refused prompt does not succeed on a fallback.
Log and monitor the decision
- Per-request record: requested alias, selected model and version, provider, region, routing reason (primary, fallback, failover), attempt count, tokens and cost, caller identity.
- Model mix baselines: alert when the share of traffic to a model, provider or region moves outside its normal band, or when any request reaches a model not on the register.
- Configuration change alerts: router configuration, pool membership and router version changes go through change control and raise an alert when made outside it.
Decision criteria
| Factor | Favours single model | Favours routing |
|---|---|---|
| Data residency | Strict single-region or sovereign requirement, or no compliant second provider | Several providers or regions all meeting the requirement, enforced by allowlist |
| Consistency | Regulated decisions, auditability, reproducible outputs | Tolerant tasks such as drafting, summarising and classification with human review |
| Tools and autonomy | Agents with write or send tools | Read-only or no tools |
| Cost pressure | Low volume or fixed budget | High volume with a wide spread of request difficulty |
| Availability | Single provider SLA is acceptable | Outage tolerance must exceed one provider's |
| Evaluation capacity | Team can evaluate one model well | Team can evaluate every pool model on each change |
- Small organisation: pin one model per use case and add a single pre-evaluated failover model in the same provider and region. Routing across providers adds evaluation and contract work that small teams rarely sustain.
- Large organisation: run routing through a central gateway with an approved model register, pools per data class, and pinned models for agents and regulated workloads. Platform teams own the register; product teams choose a pool, not a model.
- Hybrid is the norm: most mature estates route low-risk traffic and pin high-risk traffic behind the same gateway.
Worked scenario: routing a customer assistant
A European retailer wants to cut inference cost on its customer assistant, which answers product questions and, for signed-in users, looks up orders that contain personal data.
- Classify traffic: anonymous product questions (no personal data, no tools) and signed-in order queries (personal data, read tool).
- Design pools: product questions go to a two-model pool in an EU data zone, with a cheaper model for simple prompts. Order queries are pinned to one model version in the EU region with no cross-provider fallback; a second deployment of the same model in another EU region is the only failover.
- Constrain fallbacks: content-policy fallback is disabled on both routes; a refusal is returned as a refusal.
- Evaluate: the system suite runs on each pool model alone and through the router, including routing-steering cases. One pool model follows a product-page injection that the other resists, so injection-carrying retrieval content is filtered before routing and the case joins the gate set.
- Log and alert: every response record carries selected model, region and routing reason. An alert fires if any order query reaches a model other than the pinned version, or if the share of the expensive model on product questions rises sharply.
- Roll out: 5% of product traffic for two weeks with model mix and refusal rates compared against the single-model baseline, then a staged increase. Rollback is a configuration change back to the single model.
Where identity fits
A router is an identity broker: callers authenticate to it with one key and it authenticates to each provider with another. Per-key model lists turn routing policy into authorisation policy, so give each application and agent its own router key restricted to its pool, rather than one shared key with access to every model. Rotate upstream provider keys at the provider when the router is suspected compromised. See the Model Aggregator and LLM Router Key Security guide and the LLMjacking Guide.
Detection and measurement
- Off-register requests: count of requests served by any model, provider or region not on the approved register; target zero.
- Fallback rate by reason: share of requests answered by a fallback, split by error, context and content-policy reasons; a rising content-policy share needs review.
- Pool evaluation currency: share of pool models evaluated against the current system suite version; target 100% before any pool change goes live.
- Model mix drift: deviation of each model's traffic share from its baseline band, as an early signal of steering or misconfiguration.
- Selected-model log coverage: share of inference records carrying selected model, provider and region; target 100%.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev. 5 | SA-9 External System Services; AC-4 Information Flow Enforcement; CM-2 Baseline Configuration; CM-3 Configuration Change Control; AU-12 Audit Record Generation | Provider oversight, routing as flow enforcement, approved pool as baseline, pool changes under change control, selected-model logging |
| NIST CSF 2.0 | GV.SC-05; GV.SC-07; DE.CM-06; PR.PS-04 | Contract terms with model providers, supplier risk, monitoring external provider activity, log generation |
| NIST AI 600-1 | MS-2.3-001; MS-2.5-001 | Baseline performance per candidate model; no extrapolation from narrow tests to the whole pool |
| OWASP Top 10 for LLM Applications 2025 | LLM03 Supply Chain; LLM10 Unbounded Consumption | Third-party models in the pool; cost inflation through routing |
| MITRE ATLAS | AML.T0034 Cost Harvesting; AML.T0054 LLM Jailbreak; AML.T0029 Denial of AI Service | Routing-driven cost abuse, downgrade to jailbreak-prone models, availability |
| ISO/IEC 27001:2022 | Annex A 5.19 to 5.22 supplier relationships; 8.32 Change management | Model providers as suppliers; pool changes as changes |
Common failure patterns
- Ignore list instead of allowlist: new models and providers join the pool automatically because only known exclusions were configured.
- Default fallbacks left on: a regional, zero-retention primary fails over to a global endpoint during an incident, and nobody notices until an audit.
- Evaluating the favourite model only: the release suite ran on the primary; the cheaper model that answers most traffic was never tested.
- Logging the alias: logs show "chat-default", so an investigation cannot tell which model produced a harmful answer.
- Auto-update on the router version: the pool changes with the provider's release, outside change control.
- One router key for everything: a leaked key reaches every model and every budget.
Practitioner checklist
- Maintain an approved model register with model, version, provider, region and data terms, and make the router configuration match it.
- Configure pools as allowlists; never rely on exclusion lists alone.
- Restrict fallbacks to the approved set with equal or stricter residency and retention; fail closed otherwise.
- Disable or constrain content-policy fallbacks on user-facing routes.
- Pin agents with write tools and regulated workloads to a single model version.
- Run the system evaluation suite on every pool model and through the router, including routing-steering cases.
- Log selected model, provider, region and routing reason on every request; alert on off-register selections and model mix drift.
- Freeze router versions or inclusion lists and handle pool changes through change control.
- Issue per-application router keys restricted to their pool, with budgets.
Standards and references
- arXiv: Rerouting LLM Routers (Shafran, Schuster, Ristenpart, Shmatikov)
- arXiv: RouteLLM, Learning to Route LLMs with Preference Data
- OWASP: Top 10 for LLM Applications 2025
- NIST: AI 600-1 Generative Artificial Intelligence Profile
- AWS: Amazon Bedrock Intelligent Prompt Routing
- Microsoft Learn: Model Router in Microsoft Foundry
- LiteLLM: Proxy Fallbacks and Reliability
- OpenRouter: Provider Routing
Related NHI Mgmt Group resources: AI Gateway and LLM Gateway Guide · Model Aggregator and LLM Router Key Security · Data Sovereignty and Residency Guide · AI Model Risk Management Guide · Model-Level vs System-Level AI Evaluation