Major Incident Management Process Guide

Most organisations run two incident processes, one in IT operations and one in security, and the worst incidents are the ones that start in the first and turn out to belong to the second. A database cluster that falls over, a wave of failed sign-ins or a mass encryption of file shares all look like outages for the first half hour. Major incident management is the discipline that takes over when normal ticket handling is not enough: it declares the incident, puts one person in command, runs a bridge, keeps stakeholders informed and hands off cleanly to problem management and post-incident review. This guide sets out that process, including the decision point at which an outage is escalated into the security incident process and how the two run side by side. It is written for service delivery managers, incident managers, SRE leads and security operations teams who need one playbook that both sides recognise.

Key takeaways

Declare early and on clear criteria. Google's SRE guidance declares an incident if a second team is needed, the outage is customer visible or the issue is unsolved after an hour of concentrated analysis; declaring late costs more than standing down.
One commander, who does not fix things. The incident commander owns status, decisions and delegation; PagerDuty's documentation is explicit that the commander is not a resolver and that the commander's decisions are final for the duration.
Security escalation is a branch, not a restart. When indicators of compromise appear, the security incident lead joins or takes command, the communication channel moves to a private one and evidence handling changes; the bridge structure stays.
Communications run on a clock. A single status source, a stated time for the next update and pre-approved templates prevent the vacuum that drives escalations and speculation.
The incident is not over at restoration. A dated handover to problem management and a blameless post-incident review within days are what stop the same outage recurring.

What makes an incident major

An incident becomes major when its impact or urgency needs coordination beyond the normal support queue. The definition should be written down, measurable and agreed with the business, because arguing about severity during an outage wastes time. PagerDuty's public response documentation uses numbered SEV levels, treats anything above SEV-3 as a major incident and adds two rules worth copying: if unsure between two levels, treat it as the higher one, and an incident commander may trigger the full process for a lower severity incident that needs coordinated response.

Severity	Example criteria	Response	Comms cadence
SEV-1 (major)	Critical service down for many customers, contractual SLA breached, or a confirmed exposure of customer data	Incident commander paged at once, full bridge, executives informed, security lead engaged if any compromise indicator	Status page and stakeholder update at least hourly, next update time always stated
SEV-2 (major)	Core function severely degraded for most users, or incident response or monitoring capability impaired	Incident commander and bridge; executives informed on request	Hourly while customers are affected
SEV-3	Partial degradation with a workaround, limited customer impact	Owning team handles; commander can upgrade	Ticket and team channel
SEV-4 and below	Minor fault, cosmetic, single user	Normal incident queue	Ticket updates

Two rules prevent the common trap of labelling everything a SEV-1: tie each level to observable impact (customers, revenue, safety, data) rather than to how loud the reporter is, and review severity assignment in every post-incident review. Smaller organisations can collapse this to two levels, "major" and "normal", as long as the trigger is written down.

Roles on a major incident

The role model below combines Google SRE's incident command structure with PagerDuty's published roles. Roles are functions, not people: on a small incident one person may cover the deputy, scribe and internal liaison roles, while a large incident may split work into sub-teams with their own leads.

Role	Responsibilities	Must not
Incident commander (major incident manager)	Single source of truth for status; sets priorities; delegates every repair action; approves external communications; decides escalation and stand-down; owns the live incident document	Debug or make changes personally; let the bridge drift into parallel discussions
Deputy	Backs up the commander, pages missing teams, tracks time-boxed actions; must be qualified to take command	Run a second line of decisions
Operations lead and subject matter experts	Diagnose and make changes; in Google SRE's model the operations team is the only group changing the system during the incident	Make unannounced changes outside the bridge
Scribe	Keeps the timeline with timestamps, decisions, actions and links to the data behind each entry	Edit the timeline after the fact
Communications lead (customer liaison)	Drafts status page and customer updates for the commander's approval; keeps to the update clock	Publish unapproved statements, speculate on cause
Internal liaison	Briefs executives, service desk and support; mobilises additional internal responders	Pull responders off the bridge for briefings
Security incident lead (when escalated)	Assesses compromise, directs containment and evidence preservation, engages legal and privacy	Be added without the commander knowing

PagerDuty's documentation notes that the commander becomes the highest ranking person on the call regardless of day-to-day rank. Agree that principle with senior leaders in advance; executives who join a bridge and start directing engineers are a frequent cause of lost time. For the commander role in depth, see the Incident Commander Role Guide.

The major incident process step by step

Detect and declare. Monitoring, the service desk or an engineer raises the incident. Anyone on the rota may declare a major incident against the written criteria; nobody needs permission. Record the declaration time: it starts every clock that follows.
Page the commander and open the bridge. One voice bridge or video call plus one chat channel, named to a convention (for example the incident number). Post who holds which role in the channel at once.
Start the live incident document. Google SRE calls this the commander's most important responsibility: a shared, templated document with current status, impact, actions in flight and next update time at the top.
Assess impact and set priorities. The SRE order is stop the bleeding, restore service, then preserve evidence for root cause. In a suspected security incident evidence preservation moves up the list (see the next section).
Delegate in time boxes. Give each action an owner and a check-in time, and spin up sub-teams with a single lead when the bridge becomes too large to control.
Communicate on the clock. First external acknowledgement within the target set for the severity, then updates at the stated cadence, including "no new information" updates.
Control change. Emergency changes are approved by the commander and recorded by the scribe, so that every divergence from normal can be reverted later. See the Change Management Security guide.
Hand off explicitly. At shift change the outgoing commander briefs the incoming one, waits for acknowledgement and announces the handover to everyone on the bridge.
Restore, verify and stand down. Confirm recovery from monitoring and from customers' perspective, release non-essential responders, publish a resolution notice and schedule the review.
Hand over to problem management. Open a problem record linked to the incident with the timeline, suspected causes and the workarounds still in place.

When an outage becomes a security incident

The handover between operations and security is where joint processes most often fail: either the security team hears about a compromise hours late, or every outage is escalated and security stops answering. Write explicit triggers and make escalation cheap. PagerDuty's security response guidance says to trigger the process anyway when unsure and let the commander decide.

Escalation triggers

Unexplained change: configuration, accounts, firewall rules or code changed with no matching change record.
Identity anomalies: new privileged accounts, MFA resets, unexpected service principal or API key use, or a surge of failed authentications.
Data signals: files encrypted or renamed in bulk, large unexplained egress, deleted backups or snapshots.
Security tooling affected: endpoint agents stopped, logging disabled or alerts silenced alongside the outage.
External claims: a ransom note, an extortion email, a researcher report or a threat intelligence alert matching the affected service.

What changes after escalation

Aspect	IT major incident	After security escalation
Command	Major incident manager	Security incident lead takes command or joint command; major incident manager continues to run service restoration as a workstream
Channel	Open bridge and channel, wide attendance	Private, need-to-know channel, out of band if collaboration tools or identity may be compromised; PagerDuty suggests an innocuous codename
Priority	Restore fast, rebuild if quicker	Contain, then restore; preserve evidence before rebuilding (PagerDuty: shut instances down rather than delete or terminate them)
Change	Commander approves emergency changes	Changes also reviewed for evidence impact; attacker access paths closed first
Communications	Status page on cause and fix	Legal and privacy review every statement; cause wording cautious until confirmed
Notification	SLA and customer contracts	Regulatory clocks may start, for example 72 hours to the supervisory authority under GDPR Article 33 for a notifiable personal data breach, and the NIS2 Article 23 early warning within 24 hours for significant incidents in scope

The security workstream follows its own playbook: see the Incident Response Guide and, where identities are involved, the Identity Incident Response Playbook. For regulatory timelines side by side, see EU Incident Reporting Clocks.

Bridges, tools and out-of-band capability

Pre-built bridges: a standing bridge number or meeting link per severity, published in the runbook, removes minutes of setup.
Out-of-band option: keep a secondary communication path that does not depend on your corporate identity provider, email or chat platform, and test it. Outages of identity, DNS or the collaboration suite take the primary bridge down with the service. See the Major IT Outage Lessons Guide for incidents where this happened.
Bridge hygiene: announce yourself on joining, mute when not speaking, take side investigations to breakout channels and report back through a lead.
Verify participants: in a suspected security incident, confirm who is on the bridge; attackers who have access to chat or email can read or join incident channels.
Emergency access: responders may need break-glass accounts when the identity provider is the failing component. Store, monitor and test them. See the Break-Glass Account Guide.
Contacts: keep an offline call tree for responders, suppliers and executives. See the Incident Response Contact and Call Tree Template.

Outage communications

Atlassian's incident communication guidance gives practical rules: use a status page as the primary source of truth and point every other channel back to it, never go more than an hour without an update while customers are affected, always say when the next update will come and post "no new information" rather than go silent. Prepare templates in advance and get them approved.

Audience	What they need	Channel	Owner
Responders	Current status, actions, owners, next check-in	Bridge, incident channel, live document	Commander and scribe
Service desk and support	Approved customer wording, workaround, known impact	Internal broadcast, knowledge article	Internal liaison
Executives	Business impact, expected duration, decisions needed from them	Short written summary at fixed times	Internal liaison
All staff	What is affected, what to do, where to look	Intranet banner or email	Internal liaison
Customers	Impact, workaround, next update time	Status page, subscriptions	Communications lead, commander approves
Regulators and partners	Formal notification content and timing	Defined in contract or law	Legal or compliance, on advice from security lead

First updates should acknowledge the problem, describe known impact, promise the next update and, if data could be affected, say that this is being investigated. Avoid naming a cause until it is confirmed; an early "not a cyberattack" statement that is later reversed is far more damaging than "the cause is under investigation".

Problem management handover and post-incident review

Handover pack: the problem record gets the timeline, the live incident document, the list of emergency changes still in place and the open hypotheses on cause. Workarounds stay tracked as known errors until a permanent fix ships.
Review timing: PagerDuty schedules the postmortem meeting within 3 calendar days for SEV-1 and 5 business days for SEV-2, before memories fade.
Blameless and evidence based: each timeline entry links to the graph, log search or message that supports it; the review asks how the system allowed the failure, not who caused it.
Process questions too: how long until declaration, until the commander joined, until the first customer update; whether security was engaged at the right time; whether the bridge worked.
Actions with owners and dates: each action tracked to closure in the problem or change system. See the Post-Incident Review and Root Cause Analysis Template.

Measuring the process

Metric	What it shows	Typical target
Time from first alert to declaration	Whether triggers are clear and used	Minutes, falling over time
Time from declaration to commander on bridge	Rota health and paging	Within 15 minutes for SEV-1
Update cadence adherence	Communications discipline	No customer-affecting gap over 60 minutes
Time from first compromise indicator to security engagement	Operations to security handover	Within 30 minutes
Reviews held on time and actions closed by due date	Learning loop	100% held; over 90% of actions on time
Repeat incidents with the same root cause	Problem management effectiveness	Zero within 12 months

Targets are illustrative; set your own from your service levels and review them yearly.

Framework mapping

Process element	NIST SP 800-53 Rev. 5	NIST CSF 2.0	ISO/IEC 27001:2022 Annex A
Written major incident plan and roles	IR-8 Incident Response Plan, CP-2 Contingency Plan	ID.IM-04	5.24 Information security incident management planning and preparation
Declaration and severity	IR-4 Incident Handling	RS.MA-01, RS.MA-03	5.25 Assessment and decision on information security events
Tracking and documentation	IR-5 Incident Monitoring	RS.MA-01	5.26 Response to information security incidents
Stakeholder and regulatory communications	IR-6 Incident Reporting	RS.CO-02, RS.CO-03, RC.CO-03	5.26, 6.8 Information security event reporting
Restoration	CP-10 System Recovery and Reconstitution	RC.RP-01	5.29 Information security during disruption
Post-incident review	IR-4	ID.IM-03	5.27 Learning from information security incidents

NIST SP 800-61 Rev. 3, published in April 2025 as a CSF 2.0 Community Profile, frames incident response within overall risk management and suits organisations aligning operational and security incident handling. ITIL 4 covers the IT service side through its incident management, problem management and change enablement practices. See also the ISO/IEC 27035 Incident Management Guide.

Common failure patterns

Late declaration: teams spend hours fixing quietly to avoid "making it a major incident", and the customer-visible outage grows without communication.
The commander debugs: the most senior engineer takes command and then disappears into a terminal; nobody owns status, comms or decisions.
Two parallel bridges: IT and security run separate calls on the same event, making conflicting changes and issuing conflicting statements.
Rebuild before preserve: servers are reimaged to restore service before anyone checks for compromise, destroying the evidence that would show attacker scope.
Premature cause statements: an early public "no evidence of malicious activity" has to be withdrawn.
Review never happens: service comes back, everyone moves on, and the same latent fault recurs months later.

Practitioner checklist

Publish severity definitions tied to observable impact, with "if unsure, go higher" written in.
Staff a 24x7 incident commander rota with trained deputies, and agree with executives that the commander leads the bridge.
Pre-create bridges, channel naming, a live incident document template and approved communication templates.
Write the security escalation triggers and test the handover in a joint exercise at least twice a year.
Maintain an out-of-band communication path and offline contact list that do not depend on your identity provider.
Run a status page as the single external source of truth with a maximum one-hour update gap.
Require commander approval and scribe logging for every emergency change.
Hold a blameless review within days for every major incident and track actions to closure through problem management.
Report the six process metrics to service and security leadership monthly.

Major Incident Management: Run IT Outages and Security Incidents Together