Provider routing is the practice of directing model requests to different upstream services based on policy, cost, availability, or model fit. In an AI gateway, routing can shift traffic between direct provider access, cloud-hosted models, and fallback options without changing client configuration or developer workflows.
Expanded Definition
Provider routing is the policy-driven selection of an upstream model service for each request, often based on latency, availability, cost, tenant requirements, model capability, or risk controls. In NHI and agentic AI environments, routing is not just traffic management. It is an access decision that determines which provider receives prompts, context, and often secrets or tool outputs.
Definitions vary across vendors because some treat routing as a gateway feature, while others bundle it with failover, load balancing, or model orchestration. In practice, the term is most useful when it describes a controllable layer that can shift requests without forcing developers to rewrite client code. That distinction matters because the routing plane becomes part of the trust boundary and should be governed like any other privileged control. The NIST Cybersecurity Framework 2.0 is helpful here because it frames resilience and access governance as operational requirements, not just architecture choices.
Provider routing is commonly misunderstood as a harmless convenience feature, but the most common misapplication is treating route selection as a purely performance optimization when it is actually authorising where sensitive AI workloads and secrets flow.
Examples and Use Cases
Implementing provider routing rigorously often introduces policy complexity, requiring organisations to balance resilience and cost savings against tighter governance, auditability, and prompt handling controls.
- An AI gateway sends low-risk summarisation jobs to a lower-cost provider and reserves premium models for regulated workflows that require stronger tenant isolation.
- A fallback policy automatically shifts traffic away from a degraded model endpoint while preserving the same client interface and logging the decision path for review.
- A security team routes prompts containing customer data only to approved services after checking region, retention, and contract constraints, which aligns with NIST Cybersecurity Framework 2.0 principles for governance and resilience.
- A developer platform steers agentic workflows toward a provider that supports tool invocation, while non-agentic requests remain on a cheaper text-only model.
- When secret leakage risk rises, teams review routing alongside incidents such as JetBrains GitHub plugin token exposure and Code Formatting Tools Credential Leaks, because the route that carries a prompt may also carry credentials or context that should never reach an unapproved upstream.
Why It Matters in NHI Security
Provider routing matters because it determines which external or internal service receives non-human identity material, including API keys, tokens, certificates, and tool credentials. If routing policies are weak, a request can be sent to a provider that lacks the required retention controls, regional safeguards, or incident response maturity. That turns a convenience layer into a governance gap.
NHI Mgmt Group reports that 68% of organisations do not know how to fully address NHI risks, and routing is one place where those risks become visible because request paths, upstream trust, and credential exposure must all be managed together. In agentic environments, routing also affects blast radius: a misrouted tool call or prompt can move sensitive context into the wrong model boundary before controls have a chance to react. The practice should therefore be monitored as part of model governance, secrets governance, and third-party risk, not left to application teams alone. Provider routing often becomes operationally unavoidable only after a prompt, token, or regulated workload has already been sent to the wrong upstream, at which point the routing policy becomes evidence in the incident review.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST Zero Trust (SP 800-207) and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A1 | Routing affects which model endpoint receives agent prompts and tool actions. |
| OWASP Non-Human Identity Top 10 | NHI-03 | Provider routing can expose secrets and tokens if upstream selection is not governed. |
| NIST CSF 2.0 | GV.OC, PR.AC | Routing is part of governance, access control, and resilience decisions. |
| NIST Zero Trust (SP 800-207) | Zero Trust requires continuous verification of every access path, including routed model traffic. | |
| NIST AI RMF | AI risk management covers provider selection, third-party risk, and downstream harms. |
Assess routing decisions for risk, provenance, and impact before enabling failover or multi-provider use.
Related resources from NHI Mgmt Group
- Why is single-provider AI agent governance not enough for enterprise security?
- Why do identity provider failures matter so much in federated environments?
- How should security teams choose an enterprise sso provider for b2b SaaS?
- What breaks when a service provider relies on email address as the user key?
