AI Model Change Management vs Traditional Change Management: Adapting the CAB to Models, Prompts and Evals

Traditional change management assumes that nothing changes in production unless someone in your organisation raises a change, but an AI system can change its behaviour because a provider moved an alias, retired a version, edited a prompt template or adjusted a temperature setting, with no ticket anywhere. Change advisory boards (CABs) were built for deterministic software; large language model (LLM) applications change through prompts and model identifiers, are tested statistically, and may lose their rollback target. This guide is for change managers, platform owners and AI engineering leads adapting an existing CAB. It does not repeat the change classes and test matrix in the AI Model Change Approval Template; it explains why those adaptations are needed and how to run them.

Key takeaways

• The trigger model is different. Traditional change control reacts to changes your staff initiate; AI change control must also detect changes initiated by a provider, a platform upgrade policy or drift that nobody initiated.
• Prompts and settings are code. System prompts, few-shot examples, temperature, tool descriptions and retrieval settings change outputs as much as a model swap, so they belong in the configuration baseline.
• Evaluation replaces the test sign-off. A unit test either passes or fails; an LLM regression suite produces a score distribution, so the CAB approves against thresholds and a baseline, not a green tick.
• Rollback has an expiry date. Retired model versions cannot be restored, so the back-out plan must name a fallback that will still exist, and the retirement date itself becomes a planned change.
• Adapt the CAB, do not bypass it. Keep one change record and one approval trail; add AI-specific risk questions, evidence and owners so audit and incident teams still find every change in one place.

The short answer

Use your existing change management process for everything around an AI system: infrastructure, gateways, application code and access changes. Extend it with four AI-specific adaptations: pin model identifiers and treat any identifier, alias or upgrade-policy change as a normal change; put prompts and inference settings under version control as configuration items; require an evaluation report against a fixed baseline as the test evidence; and keep a tested fallback model as the rollback path. Then add detection for the changes nobody raised.

For the general discipline this builds on, see the Change Management Security guide and the SOC 2 CC8 Change Management Guide.

What each discipline assumes, in architectural terms

Traditional change management

Architecture in words: a configuration management database (CMDB) holds a baseline of configuration items (servers, applications, network devices, settings). A requester raises a change with impact, test evidence and a back-out plan, and an approver or CAB authorises it. NIST SP 800-53 expresses this as CM-2 Baseline Configuration, CM-3 Configuration Change Control and CM-4 Impact Analyses; ISO/IEC 27001:2022 Annex A 8.32 requires changes to information processing facilities to be subject to change management procedures.

Threat model: unauthorised changes, untested changes that cause outages, and changes that weaken a security control. Because the system is deterministic, a reviewed diff and passing tests are good evidence, and redeploying the previous build restores previous behaviour.

AI model change management

Architecture in words: an LLM application is a composite of a model identifier served by a provider or platform, a system prompt and templates, inference parameters, a retrieval pipeline (embedding model, chunking, index contents), tool and function definitions, guardrail policies, and the gateway routing rules that pick the model. Some of those items are controlled by a third party on its own schedule.

Threat model: the same unauthorised and untested change risks apply, plus three new ones. First, changes that arrive without a request: an alias moving to a newer snapshot, a platform auto-upgrade, or a version retirement that forces a move. Second, behaviour changes that no diff reveals: a one-line prompt edit can weaken refusal behaviour or expose the system prompt. Third, regressions that are statistical and task-specific, so a change can improve average quality while breaking a safety property for a minority of inputs. Research by Chen, Zaharia and Zou found that GPT-4's accuracy on one task (identifying prime versus composite numbers) fell from 84% in March 2023 to 51% in June 2023, and that both GPT-3.5 and GPT-4 made more code formatting mistakes in the later versions, which is why they called for continuous monitoring of LLM services.

Side-by-side comparison

Provider-initiated changes: what the platforms document

Part of the change calendar belongs to someone else. Each major platform publishes lifecycle terms and notice arrangements, and customer settings that control when a move happens. Platforms are listed alphabetically; the order is not a ranking. Terms were checked against each vendor's documentation in November 2026 and change often, so confirm against the current pages.

Three change management consequences follow. Upgrade policy is a configuration item: a deployment set to upgrade automatically has delegated change approval to the platform, which may be acceptable for a low-risk internal tool but should be a recorded decision. Retirement dates are planned changes: raise the replacement change when the deprecation notice arrives, not when the shutdown error appears. Pinning is not a guarantee of identical behaviour: Anthropic's documentation notes that infrastructure updates occasionally produce minor observable differences even when the model ID and weights have not changed, so detection still matters on pinned versions.

Prompts, parameters and retrieval as configuration items

Traditional CMDBs record that an application exists and which servers it runs on. They rarely record the artefacts that decide what an AI system says and does. The table lists the items to add to the baseline and why each needs change control.

Anything which can alter an output for the same input is a configuration item. The Prompt Lifecycle Management guide covers registries, review and access control for prompts in depth; the point for the CAB is that prompt changes appear in the same change calendar as model changes.

Evaluation gates instead of test sign-off

A traditional CAB asks "did the tests pass?". For an AI change that question has no single answer, because outputs vary between runs and quality is measured on a distribution. The CAB needs a different kind of evidence.

• Fixed baseline: every change is scored against the current production configuration on the same versioned evaluation set, so the CAB sees a delta, not an absolute number.
• Repeated runs: run each evaluation several times, or with a fixed seed where supported, and report a range. A difference smaller than the run-to-run variance is not evidence of improvement or regression.
• Safety properties as hard gates: quality metrics can trade off; safety properties should not. Treat new prompt injection failures, system prompt leakage, unauthorised tool calls and data disclosure as blocking, whatever the average score.

OpenAI's deprecation guidance makes the same recommendation for migrations: run representative requests against both models and compare outputs, latency and cost before switching production traffic. For building the suite, see AI security regression testing and the AI Security Evaluations Guide.

Rollback when the old version may not exist

A traditional back-out plan redeploys the previous artefact. For AI systems the plan has to account for parts you cannot redeploy.

• Prompt and parameter changes: fully reversible if versioned. Keep the previous version deployable with one configuration switch.
• Model version changes: reversible only until the previous version reaches its retirement or shutdown date. After that, the back-out target must be a different, already-evaluated model. Name it in the change record and evaluate it on the same suite.
• Embedding model changes: rolling back means keeping the old index alive in parallel until the change is accepted, because vectors from different embedding models cannot be mixed.
• Agent actions already taken: reverting the model does not undo emails sent or records changed. Gate write actions behind approval during the canary.
• Kill switch: NIST AI RMF MANAGE 2.4 calls for mechanisms to supersede, disengage or deactivate AI systems that depart from intended performance. A tested route to a fallback model or a safe degraded mode is the AI equivalent of the emergency back-out.

CAB adaptations that work

1. One record, AI fields added: add model ID, upgrade policy, prompt version, evaluation report link and fallback model to the existing change template rather than opening a separate AI tracker.
2. Standard changes for low-risk prompt edits: wording fixes that pass the automated suite can be approved by the system owner, keeping the CAB for judgement calls.
3. Normal change for any model identifier or upgrade policy change: including changing a deployment from pinned to automatic upgrades.
4. Provider notices as change inputs: route deprecation emails, service health alerts and lifecycle API changes into the change queue, with the retirement date as the deadline.
5. An AI reviewer on the board: someone who can read an evaluation report and challenge thresholds, alongside security and the service owner.
6. Drift as an incident trigger: when monitoring detects behaviour change with no matching change record, treat it as an unauthorised change investigation, as you would for an unexplained configuration change on a server.
7. Regulatory check for high-risk systems: where the EU AI Act applies, ask whether a change is a substantial modification; the template's section 4.4 covers the question to put to legal.

Decision criteria: how much AI-specific control a change needs

Small organisations: one owner, a version-controlled config file, a small regression suite run in CI and pinned model IDs give most of the benefit. Large organisations: centralise model routing in a gateway so pinning and upgrade policy are enforced in one place, and integrate the evaluation report into the change tool. Regulated firms should align with the AI Model Risk Management Guide.

Worked scenario: a support assistant and a retiring model

A retailer runs a customer support assistant on a pinned model version, with a retrieval index of help articles and two tools: order lookup and refund request. The provider emails a deprecation notice giving six months before shutdown.

1. The platform owner raises a normal change the same week, records the shutdown date as the deadline and names two candidate replacements.
2. Engineering runs the fixed evaluation suite five times per candidate. Candidate A scores higher on answer quality but issues refund requests without the required confirmation step in a small share of runs. That is a hard-gate failure.
3. The team adjusts the system prompt for candidate A. Because the prompt changed too, the change record now covers two configuration items and the full suite runs again, including prompt injection and system prompt leakage tests.
4. The CAB approves candidate A with candidate B evaluated and named as the fallback. Deployment goes to 5% of traffic with refund requests routed to human approval for two weeks.
5. During the canary, monitoring shows average response length up and token cost higher than baseline. The budget owner accepts the cost; the change record is updated.
6. Full rollout completes six weeks before shutdown, leaving time to revert to the old version if a late problem appears. After shutdown, the fallback becomes candidate B.

Implementing the adaptations

1. Inventory: list every AI system with its model ID, provider, upgrade policy and prompt location in your AI register. See the AI Register and AI Inventory Template.
2. Pin and record: move production systems to pinned identifiers and record any deliberate automatic-upgrade choice as a risk acceptance.
3. Version prompts and config: move prompts and parameters into a repository or registry with reviewed changes.
4. Build the baseline suite: start with 50 to 200 representative cases plus safety tests, and capture the current production scores.
5. Change the template: add the AI fields and adopt the classes in the approval template. Pilot on two systems for one change cycle.
6. Add drift detection: scheduled evaluation runs against production configuration, alerting on deviation beyond the run-to-run range.

Detection and measurement

Detection covers the changes nobody raised: compare the model identifier actually returned in API responses with the approved value, alert on gateway route or policy edits outside a change window, diff prompt registry contents against the last approved version, and run the evaluation suite on a schedule against production. Log model ID, prompt version and parameter set with every request so an incident can be tied to a configuration.

• Unrecorded change rate: configuration changes to AI systems found by detection with no change record (target: zero).
• Pinned coverage: share of production AI systems on pinned identifiers or with a recorded automatic-upgrade acceptance.
• Retirement lead time: days between a replacement going live and the retiring version's shutdown date (target: positive for every system).
• Gate effectiveness: regressions found in production that the evaluation suite should have caught.

Common failure patterns

• Calling a moving alias in production: the application uses a dateless alias or "latest" name, behaviour changes overnight, and the first sign is a customer complaint.
• Prompt edits outside change control: a product owner edits a template in a low-code console, and a refusal instruction or tool-use rule disappears with no review.
• Treating the deprecation email as information: nobody owns the notice, the shutdown date arrives, requests start failing, and the emergency change skips evaluation.
• Averages hide the regression: the CAB approves because the overall score improved, while a safety property failed on a small slice of cases.
• No viable rollback: the back-out plan names a model version that has already reached end of life, or the old embedding index was deleted.
• Separate AI process: AI changes are tracked in a team spreadsheet, so incident responders and auditors cannot see them alongside infrastructure changes.

Framework mapping

Practitioner checklist

• Every production AI system has a pinned model identifier or a recorded acceptance of automatic upgrades.
• Upgrade policy settings on platform deployments are reviewed and held in configuration under change control.
• Prompts, templates and inference parameters are versioned and changed through review.
• The change template carries model ID, prompt version, evaluation report and named fallback fields.
• A fixed evaluation suite with safety hard gates runs for every model and prompt change, with repeated runs.
• Provider deprecation notices land in the change queue with the shutdown date as the deadline.
• Every rollback plan names a target that will still exist after the change window.
• Requests log model ID and prompt version, and scheduled evaluations detect drift on pinned versions.

Standards and references

• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• NIST: AI Risk Management Framework Core
• Amazon Web Services: Amazon Bedrock model lifecycle
• Anthropic: Model IDs and versioning
• Google Cloud: Model versions and lifecycle
• Microsoft Learn: Azure OpenAI model versions
• OpenAI: Deprecations
• Chen, Zaharia and Zou: How is ChatGPT's behavior changing over time?

Related NHI Mgmt Group resources: AI Model Change Approval Template · Prompt Lifecycle Management · AI security regression testing · AI Gateway and LLM Gateway Guide · Change Management Security