TL;DR: Switching LLM providers can fail even when the API call works, because code, prompts, and evaluation history often stay tied to one model, according to Braintrust. The practical lesson is that portability depends on decoupling routing from behavior testing and preserving reusable datasets for future decisions.
At a glance
What this is: Braintrust argues that LLM provider lock-in is usually a portability problem across code, behaviour, and evaluation data, not just an API problem.
Why it matters: IAM and security teams building AI-enabled services need portable routing, testing, and traceable evaluation history so model changes do not break controls, workflows, or governance.
Context
LLM provider lock-in is not limited to whether a client can send a request to another model. In practice, teams often discover that application logic, prompt tuning, and quality evidence have all accumulated around the first provider they adopted, which makes migration harder than a simple endpoint swap.
For identity and AI governance teams, this matters because model switching changes behaviour, not just infrastructure. If a model is part of a decision workflow, a support workflow, or an agentic system, then portability has to preserve testing evidence, output consistency, and traceability across providers.
Key questions
Q: How should teams reduce the risk of LLM provider lock-in?
A: Teams should reduce lock-in by moving provider-specific logic out of application code and into an AI gateway or abstraction layer. That lets routing, fallback, and authentication change without rewriting the application. The real goal is not vendor independence in theory, but operational independence when a provider changes access, pricing, or availability.
Q: Why do LLM migrations fail even when the API connection works?
A: Because a working API call proves only that the request reached the new model. It does not prove that formatting, refusal behaviour, tool use, or extraction quality still meet production requirements. The failure is usually behavioural and evidential, not just technical, which is why shared evaluation data matters.
Q: What breaks when evaluation history is tied to one provider?
A: Migration teams lose the regression cases and baselines needed to judge whether a candidate model is safe. That means the next model choice must be rebuilt from scratch instead of compared against an existing evidence trail. In practice, this slows switching and weakens confidence in the decision.
Q: What is the difference between model routing and model portability?
A: Routing is the ability to send traffic to another model through a common interface. Portability is broader: the code, prompts, datasets, and evaluation methods must remain usable after the switch. A system can route successfully and still be non-portable if its quality evidence cannot travel with it.
Technical breakdown
Why an OpenAI-compatible gateway only solves code coupling
An OpenAI-compatible gateway standardises the request and response path so a client can talk to multiple providers through one interface. That reduces provider-specific SDK dependencies and lowers the number of application call sites that need to change during a migration. But the gateway only covers the transport and API layer. It does not guarantee equivalent model behaviour, tool-calling patterns, or output structure. The practical limit is simple: if the app is coupled to prompt shape, response parsing, or provider-specific features, the gateway removes friction but not lock-in.
Practical implication: Treat gateway compatibility as a routing control, not a portability guarantee.
Why prompts create behavioural lock-in across models
Prompts are tuned over time to a model’s formatting, refusal style, verbosity, and tool-use tendencies. That tuning is often invisible until the same prompt is executed against a candidate model and the output shifts enough to break downstream logic. This is a form of behavioural coupling, where quality depends on the combination of prompt and model rather than the prompt alone. In regulated or workflow-dependent systems, small regressions in JSON validity, extraction accuracy, or tool arguments can be operationally material even when average quality looks stable.
Practical implication: Revalidate prompts against each candidate model with the same datasets and scorers before any cutover.
Why reusable evaluation data is the real portability layer
Evaluation history becomes part of the dependency stack once production traces, labelled examples, and regression cases live inside one provider’s tooling. If that data cannot be exported and reused, the next migration starts with a weak baseline and little evidence for comparison. Portable evaluation means keeping datasets, expected outputs, metadata, and scorer definitions independent of the model. That lets teams compare providers on the same evidence and preserve failures for future regression testing. In governance terms, the evaluation record is as important as the model choice itself.
Practical implication: Store production examples and regression cases in provider-neutral datasets that can support future model reviews.
NHI Mgmt Group analysis
Provider lock-in is an evaluation governance problem before it is a platform problem: once teams cannot reuse the evidence that justified a model choice, switching becomes an exercise in guesswork. The article shows that code portability alone is insufficient because behaviour and data dependencies carry the real migration cost. For practitioners, the control question is whether each model decision can be re-tested against the same evidence base.
LLM portability now depends on preserving model lineage and regression history: a candidate model that reaches the same endpoint can still fail on format compliance, tool use, or edge-case handling. That makes evaluation datasets part of the operational control plane, not an archive. The practical conclusion is that model governance must keep test cases, scores, and failure modes available independent of any single provider.
Behavioural drift is the hidden failure mode in model switching: prompt tuning often embeds assumptions about response shape, refusal style, and tool arguments that only surface after migration. This is especially relevant where LLMs support workflow automation or agentic behaviour, because downstream systems may treat small output shifts as valid input. Practitioners should govern prompt portability with the same discipline they apply to code portability.
OpenAI-compatible routing is necessary but not sufficient for AI governance: it reduces client coupling, yet it does not answer whether the new model meets production thresholds for accuracy, compliance, and reliability. That distinction matters for identity-connected AI services because model output can influence access decisions, case handling, or automated actions. Teams need portability controls that span routing, evaluation, and approval, not just inference transport.
From our research library:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
- Read next: AI Agent Observability, Audit and Incident Response Guide
What this signals
Model portability is becoming a governance requirement for AI programmes: organisations that cannot move code, prompts, and test evidence together will treat every provider decision as a reimplementation. That creates avoidable operational friction and makes change control weaker than the surrounding application demands.
Portable evaluation data is the control that matters most: if model comparisons cannot be repeated on the same examples, the programme cannot distinguish real improvement from prompt luck or benchmark noise. The practical effect is that AI governance must manage datasets and scorers as durable assets, not disposable test artefacts.
For practitioners
- Separate routing from quality validation Use a shared client interface for inference, but require a side-by-side experiment before any provider change reaches production traffic.
- Keep evaluation datasets provider-neutral Store prompts, expected outputs, labels, and regression examples outside any single provider so they can be reused after a migration.
- Test behavioural drift on your own data Run current and candidate models against the same production examples, edge cases, and scorer definitions, then review threshold failures before cutover.
- Preserve model decision history for future reviews Retain experiment results and failure cases so the next provider review starts with a baseline instead of a rebuild.
Key takeaways
- LLM provider lock-in usually appears as a portability gap across code, prompts, and evaluation history, not as a single API integration failure.
- A shared gateway can simplify routing, but it does not prove that a replacement model preserves formatting, tool use, or extraction quality.
- Teams need provider-neutral datasets and repeatable experiments so future model decisions can be judged against the same evidence baseline.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF, NIST CSF 2.0 and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | MANAGE — AI Risk Management | Model switching depends on managing quality, validation, and change risk across providers. |
| MEASURE — AI Measurement and Monitoring | The article centres on using shared datasets and scorers to compare model behaviour. | |
| GOVERN — AI Governance and Accountability | Provider choice and migration evidence need ownership and review, not ad hoc model swaps. | |
| Recommendation — Manage model changes with repeatable evaluation, approval thresholds, and documented release decisions. Measure candidate models against the same datasets and scoring criteria before moving traffic. Assign ownership for model selection, regression evidence, and rollback approval. | ||
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy | Provider lock-in creates migration risk that should be formally managed as part of the programme. |
| PR.DS-10 — Data in Transit is Protected | Gateway routing and model switching depend on controlled transmission of requests and traces. | |
| Recommendation — Include model portability and vendor switch risk in the organisation's AI risk strategy. Protect model traffic and trace data as it moves through shared inference interfaces. | ||
| NIST AI 600-1 | Evaluation and Monitoring of Generative AI Systems | The article is fundamentally about evaluation reuse and candidate-model validation. |
| Recommendation — Keep evaluation datasets and regression cases reusable across model versions and providers. | ||
Key terms
- Model portability: Model portability is the ability to move a workflow from one AI model to another without redesigning the surrounding access, audit, or policy controls. It matters because portability is what turns a model into a replaceable component rather than a single point of failure.
- Behavioral Coupling: Behavioral coupling is the hidden dependency between a prompt and one model's output habits, such as formatting, refusal style, or tool-call behaviour. A new model can accept the same prompt and still break downstream workflows by changing the shape or consistency of the response.
- Evaluation Dataset: An evaluation dataset is a structured set of representative cases used to test whether an agent behaves acceptably before or after release. It helps teams measure output quality, tool selection, escalation behaviour, and consistency so that production decisions are based on repeatable evidence rather than intuition.
- Openai-compatible Gateway: An OpenAI-compatible gateway is an abstraction layer that presents a shared client interface while routing requests to multiple model providers. It reduces code-level dependency, but it does not by itself guarantee behavioural compatibility or evidence reuse.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps security and identity teams build the governance discipline needed when AI systems change faster than their control environment.
Published by the NHIMG editorial team on August 18, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org