Incident Management vs Problem Management: Restoring Service vs Removing Root Cause

Security teams that run every repeat compromise and every expired-certificate outage as a fresh incident pay for the same cause many times over. Incident management and problem management are separate practices with different goals, owners, clocks and metrics, and the costly mistakes happen at the boundary between them: the incident closes, the cause stays open, and nobody holds it. This guide compares the two practices for security work, sets out exactly when an incident should spawn a problem record, who owns what after the handoff, and how to measure each without one set of numbers hiding the other. It goes deeper than the mechanics of running a problem record, which are covered in the Problem Management for Security guide. It is written for incident managers, SOC leads, IT service management owners and the identity and platform teams who usually own the fix.

Key takeaways

• Incidents restore, problems remove. Incident management is judged on how fast harm stops and service returns; problem management is judged on whether the same harm stops coming back.
• The handoff is a decision, not a meeting. Define in advance which incidents must create or link to a problem record, and record that decision on the incident before it closes.
• Ownership changes at the handoff. The incident commander owns containment; the problem record needs a control owner in the team that runs the failing control, which is often identity, PKI or platform rather than the SOC.
• Credential and certificate outages are classic problems. Each expiry or leaked key looks like a one-off, but the cause is usually missing inventory, ownership or automation that will fail again on the next cycle.
• Split the metrics. Mean time to restore belongs to incidents; repeat incident rate, problem age and workaround age belong to problems. Mixing them rewards fast firefighting of fires that never stop.

The short answer

Incident management is the practice of restoring normal operation as quickly as possible after an unplanned interruption or a reduction in service quality, or in security terms after a confirmed or suspected compromise. Problem management is the practice of finding and removing the causes of incidents, or managing them when they cannot yet be removed. In the ITIL vocabulary that most service desks use, an incident is an unplanned interruption to a service or a reduction in its quality, a problem is a cause or potential cause of one or more incidents, and a known error is a problem that has been analysed but not yet resolved.

The NCSC Cyber Assessment Framework makes the same distinction in security language. Principle D2 asks organisations to address the root cause or identify systemic problems rather than fix one narrow issue, and gives the example of improving the overall patch management process rather than applying the single missing patch. Incident response fixes the instance; problem management fixes the process that produced it.

Incident management in depth

Threat model: the risk incident management addresses is time. Every hour of attacker access, data exfiltration or service outage increases harm, so the practice is built to make good decisions fast with incomplete information. It accepts temporary measures (disable an account, block a range, roll back a release) because they buy time.

Architecture in words: a detection source raises an alert; triage validates it (CSF 2.0 RS.MA-02, incident reports are triaged and validated); an incident is declared with a severity; an incident commander runs a bridge with technical leads, communications and, where needed, legal and privacy; containment, eradication and recovery follow, with emergency changes logged against the incident; a post-incident review captures the timeline and actions. The roles and bridge mechanics are covered in the Major Incident Management guide and the overall life cycle in the Incident Response Guide.

Failure modes:

• Analysis under pressure: responders try to find the deep cause during containment and slow restoration down. Root cause work belongs after the bleeding stops.
• Closure equals cure: the ticket closes when service returns, and the review actions sit in a document nobody tracks.
• Emergency changes become permanent: a firewall exception or a manual token revocation step added at 02:00 is still in place a year later, undocumented.
• Severity drives learning: only severe incidents get reviewed, which the CAF lists as a not achieved indicator under D2.b (improvement only after a severe or high-profile incident).

Controls that make it work: an incident response plan with declared severity levels (SP 800-53 IR-8), incident handling procedures (IR-4), defined post-incident review timing, and an explicit closure checklist that includes the problem handoff decision described below. PagerDuty's public response guide, for example, requires a postmortem for every SEV-1 and SEV-2 and schedules the meeting within 3 calendar days for SEV-1 and 5 business days for SEV-2.

Problem management in depth

Threat model: the risk problem management addresses is recurrence. An attacker who succeeded once through a gap will try again, and so will others using the same commodity technique. Operational causes recur on a schedule: a certificate that expired once will expire again at the next renewal date unless something changes.

Architecture in words: a problem record is opened from an incident handoff, a trend, a near miss or a test finding; analysis establishes the cause and contributing factors; a known error is recorded with a workaround and its expiry; a permanent fix is raised through change management with the problem reference; a verification period confirms that linked incidents have stopped; the record closes, and reopens automatically if a new incident links to it. Record fields and analysis methods are in the Problem Management for Security guide and are not repeated here.

Failure modes:

• The orphaned action: Google's SRE workbook notes that action items not formally tracked are often forgotten, and puts it bluntly: a postmortem without subsequent action is indistinguishable from no postmortem.
• Wrong owner: the SOC is assigned a fix to an identity, PKI or cloud control it does not run, so the item ages until someone escalates it.
• Behaviour fixes: the action is "remind staff" or "be more careful". The SRE workbook warns that actions relying on changing human behaviour are less reliable than changes to automated systems and processes.
• Problem as blame: the cause is recorded as a person. The CAF lists investigations aimed only at identifying who can be blamed as a not achieved indicator under D2.a.

Controls that make it work: a problem register with priority and age reporting, plan of action and milestones tracking (SP 800-53 CA-5 and PM-4), flaw remediation tied to the problem (SI-2), configuration change control (CM-3) for the permanent fix, and senior reporting, which CAF D2.b expects: analysis goes to senior management and feeds risk management.

The handoff decision: when an incident becomes a problem

The handoff is the point where most organisations lose the cause. Treat it as a required field on incident closure, with three permitted values: linked to an existing problem, new problem opened, or no problem required with a reason. Agree the criteria before incidents happen; the Google SRE book makes the same point for postmortem triggers, which it defines in advance (user-visible downtime over a threshold, any data loss, on-call intervention such as a rollback, resolution time over a threshold, and a monitoring failure) and allows any stakeholder to request.

What moves with the handoff: the timeline, the list of emergency changes still in place, the workarounds in use, open hypotheses on cause, and the affected identities and assets. The Post-Incident Review and Root Cause Analysis Template is the right place to capture these; the problem record links to the review rather than copying it.

What does not move: containment authority. If the threat is still active, the incident stays open. Opening a problem must never be used to close an incident early.

Ownership split after the handoff

Map these roles into your existing Incident Response RACI Template so the split is written down before the next incident rather than negotiated during it.

Recurring credential and certificate outages as problems

Non-human identity failures are the clearest case for problem management because their causes are scheduled. A certificate has a fixed expiry date; a client secret or API key has a rotation date; a service account password policy forces a change on a cycle. If one instance failed because nobody owned it, every other instance with the same ownership gap is a future incident.

• Expired TLS certificates: the incident is the outage; the problem is usually missing inventory, no named owner, or renewal that is automated on one host but never deployed to the load balancer or appliance. Public TLS lifetimes fell to 200 days from 15 March 2026 and are scheduled to fall to 100 days in March 2027 and 47 days in March 2029, so a manual renewal step will fail several times a year. See the Certificate Expiry Outages and 47-Day TLS Certificates Guide.
• Expired client secrets and app credentials: an integration stops working on the day a secret expires. The incident team rotates it by hand; the problem is the lack of an expiry inventory, alerting and owner per application.
• Leaked or stale credentials: each leaked key is an incident, but repeated leaks point at a cause such as long-lived keys, no secret scanning or no rotation. In the Snowflake Breach, Mandiant reported that attackers used credentials harvested by infostealers, some dating back to November 2020, against accounts without MFA, rotation or network allow lists. Handled one customer at a time, that is hundreds of incidents; handled as a problem, it is three missing controls.
• Rotation that breaks dependants: a rotated service account password takes down jobs that hard-code it. The problem is undocumented dependencies, not the rotation itself.

Worked scenario: an internal API gateway fails at 07:10 because its certificate expired. The incident commander declares SEV-2, the platform on-call issues and installs a new certificate, and service returns at 08:05. At closure, the handoff field triggers because the cause is an expiry failure. A search of the problem register finds an existing problem opened four months earlier for an expired certificate on a different appliance, still in known error state with a workaround of "calendar reminder per certificate". The incident links to it, which raises its priority automatically on the second linked incident. The problem manager assigns the PKI owner as control owner; the permanent fix is an inventory of every certificate on load balancers and gateways, ACME-based renewal where the device supports it and monitoring of the served certificate where it does not. The problem closes only after a full renewal cycle passes with no linked incident. Related records: the Certificate Inventory Register Template and the Certificate Renewal Runbook Template.

Metrics: keep the two scorecards separate

A single dashboard that blends both practices usually rewards the wrong thing. A team that restores a recurring outage in 20 minutes every month looks excellent on time to restore and terrible on recurrence, and only separate scorecards show both.

For the wider security operations metric set and how time to detect and time to respond are defined, see the Security Operations Metrics Guide.

When to lean on each, and the trade-offs

• Small teams: the same person will often be incident commander and problem manager. Keep the records separate anyway; the separation is what stops the cause closing with the incident. A monthly 30-minute problem review is enough to start.
• Large or regulated organisations: appoint a problem manager role distinct from incident command, report problem age and repeat rate to the security steering group, and record long-running known errors in the risk register so acceptance is explicit.
• High incident volume, low severity: problem management pays off most here, because clusters of low-severity incidents (phishing, token theft, misconfigured storage) usually share a handful of causes.
• Novel, severe incidents: incident management dominates until containment; do not let root cause debates slow recovery.
• Trade-off: problem records cost analyst and owner time and can become a backlog nobody reads. Use clear triggers and close "no systemic cause" problems quickly rather than opening problems for everything.
• Trade-off: a fast permanent fix may need a normal change with testing, which takes longer than the emergency workaround. Accept the delay, but put an expiry on the workaround.

Where identity fits

A large share of security problems sit with identity owners: long-lived API keys, service accounts without owners, certificates without inventory, help desk resets without strong verification and OAuth grants nobody reviews. They recur because the incident team can contain each instance but cannot change the identity control. Name the identity control owner on the problem record, and link the relevant standard, for example the Identity Incident Response Playbook for containment steps and the Service Account Security Guide for the permanent fix to repeated service account misuse.

How to implement

1. Agree definitions: adopt incident, problem, known error and workaround in the incident response plan and the service management tool, with security categories in both.
2. Add the handoff field: make the problem decision mandatory on closure of every security incident above the lowest severity, with the three permitted values and a reason for "no problem required".
3. Publish triggers: adopt the trigger table above, adjusted to your severity scale, and include expiry and rotation failures as an automatic trigger.
4. Assign roles: name a problem manager and a control owner for each major control family (identity, PKI, endpoint, cloud, email).
5. Pilot: run the handoff for one quarter on high-severity incidents and all certificate and credential outages, then extend to recurring low-severity clusters.
6. Backfill: cluster the last 12 months of incidents by cause and open problems for the top five clusters.
7. Report: separate incident and problem scorecards monthly; problem age and repeat rate go to the steering group.
8. Roll back safely: if the handoff field creates noise, narrow the triggers rather than removing the field; the decision on closure is the control.

Framework mapping

Common failure patterns

• Problem closed with the incident: the tool or team closes everything when service returns; the cause has no record.
• Handoff by meeting only: the review discusses causes but no problem record is created or linked, so actions are lost.
• Expiry treated as bad luck: each expired certificate or secret is logged as a separate incident and the missing inventory is never addressed.
• Same team, same metric: time to restore is the only number reported, so the team is rewarded for fast repeats.
• Problem owned by the SOC for a control it does not run: the fix waits for an identity or platform team that was never assigned it.
• Repeated credential incidents handled one at a time: as in the Snowflake Breach, where missing MFA, rotation and allow lists were the shared cause behind many account compromises.

Decision checklist

• Is the threat still active? If yes, stay in incident management.
• Has this cause, technique or control failure appeared before? Link to the existing problem.
• Was a workaround or emergency change used that is still in place? Open or link a problem.
• Did monitoring fail to detect it? Open a problem against detection.
• Was the cause an expiry, rotation or ownership gap? Open or link a problem, always.
• Does the fix belong to a team outside the SOC? Name that control owner on the problem.

Practitioner checklist

• Keep incident and problem records separate, linked in both directions.
• Make the problem handoff decision a mandatory incident closure field with a reason.
• Publish handoff triggers, including expiry and rotation failures and detection failures.
• Name a control owner outside the SOC where the failing control lives elsewhere.
• Give every workaround an expiry date and track it on the problem.
• Route permanent fixes through normal change with the problem reference.
• Close problems only after a verification period with no linked incidents.
• Report incident and problem metrics on separate scorecards.

Standards and references

• NCSC: CAF Principle D2 Lessons learned
• NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• NIST: Cybersecurity Framework 2.0
• NIST: SP 800-53 Rev. 5
• ISO: ISO/IEC 20000-1:2018 Service management system requirements
• Google: Site Reliability Engineering, Postmortem Culture
• Google: The Site Reliability Workbook, Postmortem Culture
• PagerDuty: Incident Response Guide, Postmortem Process

Related NHI Mgmt Group resources: Problem Management for Security · Major Incident Management · Post-Incident Review and Root Cause Analysis Template · ITIL 4 Information Security Management Guide