When a business process depends on a hosted model, a provider outage, a regional capacity squeeze or an exhausted rate limit stops that process as surely as a database failure, yet most continuity plans still list the model API as "the vendor's problem". This guide covers continuity planning for applications and agents built on large language model (LLM) providers: classifying which AI-dependent processes need a fallback, the failure modes that matter (outage, throttling, spend caps, model retirement, behaviour change), what each major provider documents about capacity and retries, how to qualify a fallback model so it is safe to switch to, and how to test it. It is for platform owners, SRE and resilience teams, and the security and risk staff who sign off AI services. Routing mechanics inside a gateway are covered in the AI Gateway and LLM Gateway Guide, and general SaaS fallback procedures in SaaS Outage Continuity; this page covers what is specific to AI platforms.
Key takeaways
- Classify before you engineer: most AI features can degrade gracefully; only a few (customer-facing agents, fraud or triage steps in a live workflow) need a hot fallback, and those need a tested one.
- Throttling is the common failure, not total outage: 429 and overload responses under shared capacity happen far more often than a full provider outage, and the providers' own docs say retries without backoff make it worse.
- A fallback model is a change, not a spare: a different model behaves differently, so it needs the same evaluation, safety testing, data residency check and approval as the primary before it can take traffic.
- Standby credentials are non-human identities: fallback API keys or roles must exist, be scoped, be rotated and be monitored before the outage, or the switch fails at the moment it is needed.
- Not every 429 is retryable: spend caps and quota exhaustion return rate-limit style errors that never clear on retry; your runbook must tell them apart from transient throttling.
Failure modes specific to AI platforms
| Failure mode | How it shows up | Typical duration | Primary response |
|---|---|---|---|
| Provider or regional outage | 5xx errors, timeouts, status page incident | Minutes to hours | Fail over to a qualified secondary region or provider; degrade otherwise |
| Shared capacity throttling | 429 or overload responses while you are under your own limits | Seconds to hours | Backoff with jitter, shed low-priority load, use reserved capacity for critical paths |
| Own rate limit exhausted | 429 naming the limit hit (requests or tokens per minute) | Until the window refills | Smooth traffic, cache, request a limit increase in advance |
| Spend cap or quota reached | Rate-limit style error with no retry hint | Until the cap resets or is raised | Alert and escalate; retrying never helps |
| Model retirement | Deprecation notice, then errors on the old model name | Permanent | Planned migration with evaluation; track retirement dates |
| Silent behaviour change | Quality, refusal rate or output format drifts after a model update | Permanent until you adapt | Pin model versions; run regression evaluations on each change |
| Credential failure | 401 or 403 after a key is revoked, expired or leaked | Until a new credential is issued | Pre-staged standby credential; rotation runbook |
| Abuse-driven exhaustion | Traffic or token spend spike from a compromised key or runaway agent | Until stopped | Per-key budgets and kill switch; see the Denial-of-Wallet and AI Agent Cost Runaway Guide |
MITRE ATLAS lists the hostile versions of the last row as AML.T0029 Denial of AI Service and AML.T0034 Cost Harvesting, and the OWASP LLM Top 10 2025 covers them as LLM10 Unbounded Consumption. A continuity plan should treat a deliberate exhaustion attack and an accidental one the same way: both end with the service unavailable.
Classifying AI-dependent processes
Run a short business impact analysis for every production use of a hosted model. The tiers below are our recommendation; adjust the tolerances to your own impact thresholds.
| Tier | Example | Tolerable outage (ours) | Continuity pattern |
|---|---|---|---|
| 1: AI in the critical path | Customer-facing agent handling orders, claims triage step that blocks a workflow | Under 15 minutes | Hot fallback to a qualified second region or provider, plus a non-AI path |
| 2: AI assists a human | Drafting replies for support staff, code assistant, document summaries | Hours | Degrade to manual work; queue requests; clear user message |
| 3: Batch and back office | Overnight classification, enrichment, report generation | A day or more | Queue and replay; use batch interfaces; no failover |
| 4: Experimental or internal | Pilots, internal chat tools | Best effort | Status message only |
- Find the hidden tier 1 uses: agents that call models inside other workflows (ticket routing, identity verification steps, security triage) are often missed because nobody thinks of them as "the AI product".
- Record the dependency chain: provider, region, model and version, deployment type, the credential used, and any gateway in between. This is the supplier inventory that NIST CSF 2.0 ID.AM-04 and GV.SC-04 ask for.
- Small organisations: one provider with a documented degrade-to-manual path is usually proportionate. Large organisations with tier 1 uses need at least a second region, and often a second provider.
What providers document about capacity and retries
Providers are listed alphabetically. Each row states the provider's own signals, retry guidance and capacity options, from its documentation as checked in November 2026. These are customer-side design choices under shared responsibility; confirm current values in each provider's docs.
| Provider | Throttling and capacity signals | Documented retry guidance | Capacity and resilience options |
|---|---|---|---|
| Amazon Bedrock | Throttling errors; CloudTrail records the processing Region in additionalEventData.inferenceRegion for cross-Region calls | Bounded retries plus multi-Region or multi-model fallback in the application | Geographic and global cross-Region inference profiles; Provisioned Throughput |
| Anthropic API | 429 rate_limit_error naming the limit; 529 overloaded_error; anthropic-ratelimit-* headers; spend cap returns error code enforced_spend_limit_reached | SDKs retry 429 and 5xx with exponential backoff (twice by default) and honour retry-after; ramp traffic gradually to avoid acceleration limits | Priority Tier; Message Batches API for work that can wait |
| Azure OpenAI in Microsoft Foundry | 429 with retry-after-ms; x-ratelimit-* headers; a lower x-ratelimit-limit-tokens than configured indicates a temporary adjustment | Honour retry-after-ms, otherwise exponential backoff with jitter, capped retries; disable SDK retries if you add your own | Standard, Global Standard, Data Zone Standard and Provisioned (PTU) deployment types; quota per region, model and deployment type |
| Google Gemini Enterprise Agent Platform | 429 "Resource exhausted" on pay-as-you-go; "Exceeded the Provisioned Throughput" when over reserved capacity | Truncated exponential backoff; smooth spikes and ramp gradually | Global endpoint for pay-as-you-go; Provisioned Throughput with overage on demand |
| OpenAI API | 429 (including slow_down), 503 server_is_overloaded; x-ratelimit-* and Retry-After headers | Honour Retry-After plus jitter, else exponential backoff; cap attempts and total time; do not retry quota or billing errors | Usage tiers with higher limits; Batch API with its own queue limits |
Two points recur across these docs. Failed requests still count against per-minute limits, so tight retry loops extend throttling. And limits are often enforced over windows shorter than a minute, so a burst can be throttled while the minute total looks fine. For provider-specific hardening, see the Amazon Bedrock Security Guide, Anthropic API Keys Security Guide, Azure OpenAI and Azure AI Foundry Security Guide, Google Vertex AI Security Guide and OpenAI API Keys Security Guide.
Designing the fallback
Options compared
| Pattern | Protects against | Cost and effort | Main risk |
|---|---|---|---|
| Retry with backoff only | Transient throttling | Low | Does nothing in a real outage |
| Second region, same provider and model | Regional outage and regional capacity | Low to medium | Data residency if the second region is in another jurisdiction |
| Reserved capacity for critical paths | Shared capacity throttling | Medium to high (committed spend) | Paying for idle capacity; overage still goes to shared pools |
| Second provider or model | Provider-wide outage, retirement, account suspension | High (evaluation, contracts, prompts per model) | Different behaviour, safety profile and data terms |
| Self-hosted open-weight model | Provider dependency in general | High (GPU capacity, operations, patching) | Lower quality; you now run the model supply chain |
| Non-AI path | Everything | Depends on process | Staff unfamiliar with the manual procedure |
Qualifying a fallback model
- Evaluation parity: run the same evaluation set used to approve the primary and set a minimum score for the fallback. Record the result with the model version.
- Safety and guardrails: re-run prompt injection and harmful-output tests; refusal behaviour and tool-calling formats differ between models.
- Data terms: the fallback must meet the same retention, training-use and residency terms as the primary. A silent switch from a regional deployment to a global one is a data transfer you did not approve. Cross-Region routing options such as Bedrock's global profiles can process requests in any supported commercial Region, while geographic profiles stay within a geography.
- Prompt and tool compatibility: keep provider-specific prompts and tool schemas in versioned configuration so the fallback path is tested code, not a hope.
- Approval: record the fallback in the AI system inventory with the same owner and sign-off as the primary.
Where identity fits
Every fallback path needs a credential, and in an outage nobody has time to request one. Treat standby provider credentials as production non-human identities:
- Provision ahead: create the secondary provider's API key, workload identity or role in advance, scoped to the specific models and projects the fallback uses, and store it in the secrets manager the application already reads.
- Keep it alive: rotate standby credentials on the same schedule as primaries and exercise them with a low-volume synthetic call each day, so expiry or revocation is found before the outage.
- Prefer federation: where a provider is reached through your cloud (Bedrock through IAM roles, Azure OpenAI through managed identities, Google through service accounts), use those identities rather than static keys.
- Budget each key: apply spend limits per credential, so a leaked standby key cannot run up costs or exhaust shared quota. Watch each credential's spend and call pattern even when the platform is healthy: a stolen key used by someone else consumes the same quota your fallback depends on.
Implementation plan
- Inventory (weeks 1 to 2): list every production model call with provider, region, model version, deployment type, credential and tier.
- Instrument (weeks 2 to 4): log status codes, provider error types, rate-limit header values and latency per call. Alert on 429 and 5xx rates per provider and on remaining-capacity headers falling below 20%.
- Fix retries (weeks 2 to 4): one retry layer only, honouring provider retry hints, with jitter, capped attempts and a total time budget; classify spend-cap and quota errors as non-retryable.
- Build degrade paths (month 2): for tier 2 and 3, queue and replay, plus user messaging. For tier 1, add a qualified secondary region or provider behind a feature flag.
- Test (month 3, then quarterly): in a staging environment, block the primary endpoint, inject 429 and 529-style responses, and revoke the primary credential. Confirm failover time, output quality and data routing. Roll back by clearing the flag.
- Govern: track model retirement notices, review capacity reservations against actual peak use, and include AI providers in the supplier exit plan.
Detection and measurement
- Throttle rate: share of model calls returning 429 or overload responses, per provider and model, per hour.
- Failover time: time from primary failure to the fallback serving tier 1 traffic, measured in each test (target under 5 minutes, ours).
- Fallback readiness: share of tier 1 processes with a qualified, tested fallback in the last 90 days (target 100%).
- Standby credential health: successful daily synthetic calls on each standby credential (target 100%).
- Retirement lead time: days between a retirement notice and migration to the new model (target: finished before the provider's date, with evaluation evidence).
- Spend-cap headroom: percentage of monthly cap used, alerting at 70% and 90%.
Framework mapping
| Framework | Reference | How it applies |
|---|---|---|
| NIST CSF 2.0 | ID.AM-04; GV.SC-04; PR.IR-04; RC.RP-01 | Supplier service inventory, criticality, capacity for availability, executing recovery |
| NIST SP 800-53 Rev 5 | CP-2 Contingency Plan; CP-4 Contingency Plan Testing; CP-10 System Recovery and Reconstitution; SA-9 External System Services; SC-5 Denial-of-service Protection | Plan, test and recover AI-dependent services; manage provider services; resist exhaustion |
| ISO 22301 | Business impact analysis and continuity strategies | Tiering and continuity patterns; see the ISO 22301 Business Continuity Guide |
| MITRE ATLAS | AML.T0029 Denial of AI Service; AML.T0034 Cost Harvesting | Hostile exhaustion of model capacity or spend |
| OWASP LLM Top 10 2025 | LLM10 Unbounded Consumption | Limits, budgets and throttling at the application |
| EU DORA (financial entities) | ICT third-party risk and exit strategies | AI providers supporting critical functions belong in the register of information; see the DORA ICT Third-Party Risk and Register of Information Guide |
Common failure patterns
- Retry storms: SDK retries plus a custom retry library plus a gateway retry multiply the load on a throttled provider. The Azure and OpenAI docs both warn against nested retries.
- Treating a spend cap as throttling: code that retries every 429 spins on a monthly cap that will not reset until the next billing period.
- Untested fallback: the secondary provider's prompts, tool schemas or credentials were never exercised, so failover produces errors or poor output in front of customers.
- Silent residency breach: failover routes regulated data to a region or provider not covered by the data protection assessment.
- Single shared key: one API key for every application means one leaked key, one revocation or one budget breach takes all AI features down at once.
- Ignoring retirement notices: the model name in configuration stops working on the retirement date and nobody owned the migration.
Practitioner checklist
- Every production model call is inventoried with provider, region, model version, deployment type, credential and tier.
- Tier 1 processes have a qualified fallback (evaluated, safety-tested, residency-checked) and a non-AI path.
- Exactly one retry layer exists, honouring provider retry hints with jitter and a total time cap.
- Spend-cap and quota errors are classified as non-retryable and alert a named owner.
- Standby credentials exist, are scoped, rotated and exercised daily by a synthetic call.
- Throttle rate, remaining-capacity headers and spend headroom are monitored per provider.
- Failover is tested quarterly with blocked endpoints, injected errors and a revoked credential.
- Model retirement dates are tracked, with migrations planned and evaluated before the date.
Standards and references
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- MITRE ATLAS: AML.T0029 Denial of AI Service
- AWS: Amazon Bedrock cross-Region inference
- Anthropic: API rate limits
- Anthropic: API errors
- Microsoft: Manage Azure OpenAI quota
- Google Cloud: Error code 429
- OpenAI: Rate limits
Related NHI Mgmt Group resources: SaaS Outage Continuity · Third-Party Concentration Risk Guide · AI Gateway and LLM Gateway Guide · Denial-of-Wallet and AI Agent Cost Runaway Guide · Business Continuity and Disaster Recovery Guide