AI Incident Management and Escalation: Triage, Severity, Rollback and Ownership for AI Systems

Many AI incidents involve no attacker at all, so they never reach a security incident process that only triggers on compromise. A credit model that starts refusing a protected group, a support chatbot that invents a refund policy, or a vendor model update that silently changes outputs are all AI incidents, yet they often land with a product team that has no severity scale, no escalation path and no way to roll back. This guide sets out a general incident management process for AI systems of every kind, not just agents: how to recognise and classify an AI incident, how to score severity, who to escalate to, how to contain and roll back models, and how to close the loop into governance. It is for AI product owners, incident managers, model risk teams and security operations leads.

Key takeaways

• Define AI incident broadly: the OECD definition covers harm from the development, use or malfunction of an AI system, so safety, fairness and reliability failures belong in scope alongside attacks.
• Use one intake, several tracks: a single front door for AI incidents, with routing to security, model risk, privacy or legal by incident type.
• Score severity on harm, not on technology: people affected, reversibility and legal exposure drive the level; a model error can be more severe than a breach.
• Plan rollback before you need it: pinned model versions, a known-good fallback and a manual process are what make containment possible in minutes.
• Preserve the system state: under the EU AI Act, providers of high-risk systems must not alter a system in ways that affect evaluation of the cause before informing authorities.

What counts as an AI incident

The OECD defines an AI incident as an event where the development, use or malfunction of one or more AI systems directly or indirectly leads to harm: injury or harm to health, disruption of critical infrastructure, violations of human rights or breaches of law protecting fundamental, labour and intellectual property rights, or harm to property, communities or the environment. An AI hazard is an event that could plausibly lead to such an incident. Treat hazards (near misses) as reportable internally; they are the cheapest lessons available.

The NIST AI RMF expects the same breadth. MANAGE 2.3 calls for procedures to respond to and recover from previously unknown risks, MANAGE 2.4 for mechanisms and assigned responsibilities to supersede, disengage or deactivate AI systems whose outcomes are inconsistent with intended use, and MANAGE 4.3 for incidents and errors to be communicated to relevant AI actors, including affected communities, with tracking and recovery documented.

An AI incident taxonomy for routing

One incident can sit in several rows. Assign a single incident lead and bring in the other teams, rather than running parallel processes.

Severity scoring for AI incidents

Extend the organisation's existing scale (see the Incident Severity Classification Matrix Template) with AI-specific factors rather than inventing a separate one. The matrix below is our recommended starting point at NHI Mgmt Group; calibrate it to your risk appetite.

Score on these questions during triage:

• Who is affected: customers, employees, the public, and whether any are vulnerable groups.
• How many decisions or outputs: estimate from logs, not from the first report.
• Reversibility: can affected decisions be reopened and corrected?
• Use case tier: incidents in high-tier use cases start one level higher. See the AI Use Case Risk Tiering Guide.
• Legal trigger: could this be a serious incident under EU AI Act Article 3(49), a personal data breach, or a sector reporting event?

Escalation paths and roles

Write the escalation rule as a condition, not a judgement call: for example, "any suspected Article 3(49) harm goes to legal within one hour of identification". The Incident Commander Role Guide covers running the incident itself.

Containment and model rollback

NIST AI 600-1, the generative AI profile, asks for protocols so systems can be deactivated when necessary (GV-1.7-001) and for testing rollover and fallback technologies, which may include manual processing (GV-6.2-006). In practice, containment options run from least to most disruptive:

1. Tighten guardrails: add an output filter or block a topic while the cause is found. See the AI Guardrails and AI Firewalls Guide.
2. Add human review: route affected decision types to a person before they take effect.
3. Roll back: revert to the last known-good model version, prompt template, retrieval index or feature pipeline. This only works if each is versioned and the previous version is still deployable.
4. Fail over: switch to a rules-based or manual process for the affected decisions.
5. Suspend: take the use case offline and communicate the outage.

Rollback readiness checks, recommended by us:

• Version everything that changes outputs: model weights or provider model version, system prompts, retrieval sources, feature code and thresholds. A model registry gives the audit trail; see Securing the model registry itself.
• Pin provider models: call a dated model version rather than an alias that the provider moves, so a vendor change is a deliberate upgrade you can reverse.
• Keep the fallback warm: test the manual or rules-based path at least twice a year.
• Know the data dependencies: rolling back a model while keeping a new feature pipeline can cause a second incident.

Evidence and the regulatory clock

Capture before you change. Snapshot the model version, configuration, prompts, guardrail settings, input and output logs for the affected window, and monitoring dashboards. This matters legally: EU AI Act Article 73(6) says providers of high-risk AI systems must not carry out investigations that alter the system in a way that could affect later evaluation of the causes before informing the competent authorities. Article 73 also sets reporting deadlines of no later than 15 days after awareness, 10 days for a death and 2 days for widespread infringements, with an initial incomplete report allowed. Under the Digital Omnibus on AI, the high-risk rules for Annex III systems apply from 2 December 2027 (AI Act Service Desk timeline), so confirm which obligations already apply to each system. Reporting duties are covered in depth in the AI Incident Reporting Guide; the AI System Incident Report Template gives the record format.

Third-party and foundation model incidents

Many AI incidents start at a provider: a model update, a safety filter change, an outage or a provider security incident. NIST AI 600-1 suggests requesting notification and disclosure of serious incidents arising from third-party data and systems (GV-6.2-007) and aligning third-party incident response plans with the organisation's own (GV-6.2-003).

• Contract for notice: require advance notice of model retirement and material behaviour changes, and prompt notice of provider incidents affecting your data.
• Watch provider status and change logs: route them to the model owner, not a shared inbox.
• Regression test on change: run your evaluation set whenever the provider version changes, before production traffic moves.
• Rotate credentials on provider breach: API keys for model providers are non-human identities; revoke and reissue them through your secrets manager if the provider reports a compromise. See the Secrets Management Guide.

How to implement

The steps below are NHI Mgmt Group recommendations.

1. Agree the definition and taxonomy: adopt the OECD wording and the routing table above in the incident policy.
2. Link to the register: every incident record references the use case register entry, tier and owner. See the AI Governance Guide.
3. Extend the severity matrix: add the AI harm criteria and legal triggers, and get legal sign-off on escalation rules.
4. Open the intake: one form or queue for staff, customers and monitoring alerts, including near misses.
5. Prove rollback: for each high-tier system, document and test the rollback and fallback path.
6. Exercise: run a tabletop for a non-security scenario such as a biased decision model. The AI agent incident exercises page has formats you can adapt.
7. Review and feed back: after-action reviews update risk assessments, evaluation sets and guardrails, as NIST AI 600-1 GV-1.5-002 suggests.

Small organisations can run this inside the existing incident process with an AI flag and one named AI governance contact. Large or regulated organisations usually need a model risk function on call and a formal link between the incident system and the model registry.

Detection and measurement

Detection sources for non-security AI incidents include output quality monitoring, drift and fairness metrics, guardrail block rates, user feedback and complaints, override rates by human reviewers, and provider change notices. Track:

• Mean time to detect: from first harmful output to incident creation, by source.
• Mean time to contain: from incident creation to guardrail, rollback or suspension.
• Rollback readiness: share of high-tier systems with a tested rollback in the last six months.
• Near-miss ratio: hazards logged per incident; a falling ratio usually means under-reporting, not fewer problems.
• Action closure: percentage of post-incident actions closed on time.

Common failure patterns

• Security-only triggers: the incident process only opens on compromise, so fairness and accuracy failures stay in product backlogs.
• Floating model aliases: the provider moves the model under a stable name and nobody can say which version produced the harm.
• Fixing before capturing: engineers retrain or edit prompts before the state is preserved, losing the evidence regulators may need.
• No owner for the vendor relationship: provider notices go unread until customers complain.
• Correcting the model but not the decisions: outputs already given to people are never reviewed or corrected.

Framework mapping

Practitioner checklist

• Adopt a broad AI incident definition that includes harm without an attacker.
• Route AI incidents through one intake with a single incident lead.
• Add AI harm criteria and legal triggers to the severity matrix.
• Write escalation rules as conditions with named roles and times.
• Version models, prompts, retrieval sources and thresholds; pin provider model versions.
• Test rollback and manual fallback for every high-tier system.
• Snapshot system state before any fix and check reporting clocks with legal.
• Contract for provider notice of model changes and incidents.
• Log near misses and feed after-action findings into evaluation sets.

Standards and references

• NIST: AI Risk Management Framework Core
• NIST: AI 600-1 Generative Artificial Intelligence Profile
• AI Act Service Desk: Article 73 Reporting of Serious Incidents
• OECD.AI: AI Incidents Monitor methodology and definitions
• Responsible AI Collaborative: AI Incident Database
• NIST: SP 800-61 Rev 3 Incident Response Recommendations

Related NHI Mgmt Group resources: NIST AI RMF Guide, AI Agent Kill Switch and Containment Guide, AI Agent Observability and Incident Response Guide, Incident Communication Plan Template.