How to measure incident quality when detection strength varies

At a glance

What this is: This is a framework for measuring incident quality by balancing detection diversity against dependence on a single loud signal.

Why it matters: It matters because SOC and detection engineering teams need a way to judge whether incidents are resilient enough to survive a broken rule, blind spot, or noisy source.

👉 Read Expel's analysis of how to measure incident quality and detection strength

Context

Incident quality is not the same as incident severity or response speed. The problem Expel is addressing is whether an incident is well supported by multiple detections or whether it depends too heavily on one alert source, which makes it easier to miss if that control fails. In a SOC, that distinction affects detection engineering, triage confidence, and how quickly coverage gaps are exposed.

For identity-heavy environments, the same logic applies to service accounts, API keys, and other non-human identities when they trigger only one alert path. A mature detection programme should be able to show whether access abuse, lateral movement, or secret misuse is being corroborated by more than one control rather than hiding behind a single noisy signal.

Key questions

Q: What breaks when an incident relies on only one detection source?

A: The incident becomes fragile because a single broken rule, disabled sensor, or noisy data path can remove the only evidence supporting triage. When that happens, attackers can blend into the gap even if the underlying behaviour is still active. Stronger incidents are corroborated by multiple detections, so losing one source still leaves a recognisable picture.

Q: Why do repeated alerts not always mean better detection quality?

A: Repeated alerts can simply mean one rule is firing many times, not that the incident is well covered. Detection quality improves when several independent signals support the same event, because that reduces the chance that one failure hides the entire pattern. Alert count matters, but only when it is paired with diversity and balance.

Q: How can security teams tell whether incident readiness is actually improving?

A: Look for shorter containment times, fewer repeat interventions from the same weakness class, and clearer ownership for revoke and isolation actions. If incident counts stay flat while the same exposed paths keep appearing, resilience has not improved in a meaningful way.

Q: How should teams measure identity-related incidents in modern environments?

A: They should check whether identity abuse is visible across more than one control layer, such as IAM logs, endpoint telemetry, and cloud activity. NHI misuse is especially hard to spot when only one signal exists, because service accounts and tokens often behave like expected automation. Cross-source corroboration is the main guardrail.

Technical breakdown

How detection strength differs from alert volume

Alert volume alone does not tell you whether an incident is well covered. A single detection that fires many times can look strong, but it also creates a brittle dependency if all evidence comes from one control. Expel’s approach separates volume from diversity by asking how many unique detections support the incident and how much those detections contribute relative to one another. That is why two incidents can have the same total alerts but very different quality. In practice, the useful question is not how loud the incident was, but how many independent signals would still remain if one source disappeared.

Practical implication: score incidents by unique detection coverage as well as raw alert count, then tune any incident that collapses to one source.

Why entropy metrics alone can misread incident quality

Shannon diversity and Hill numbers capture proportional spread, but they can penalise incidents that have a useful amount of repetition in one detection. That matters because one detection firing multiple times may reflect a real attack pattern, not just noise. The challenge is to avoid confusing balanced diversity with operational quality. Expel’s method shows that a mathematically elegant measure can still misclassify incidents if it ignores total volume. For SOC teams, the lesson is that diversity metrics are useful, but only when paired with a measure that captures how much evidence exists, not just how evenly it is distributed.

Practical implication: do not rely on diversity scores alone when assessing detection coverage or incident robustness.

What capped counts and dominance ratio reveal about resilience

Capped counts give repeated detections extra weight without letting one loud source dominate the whole picture. Dominance ratio then measures how much the loudest detection accounts for the incident, which exposes over-reliance on a single rule or telemetry source. Together, the two measures help distinguish a robust incident from one that only looks healthy because one control fired repeatedly. This is particularly relevant in environments where a single access alert or a single endpoint rule can conceal gaps in cloud, identity, or network coverage. The technical point is simple: a resilient incident is one that still makes sense when one detection is removed.

Practical implication: use capped counts and dominance ratio to identify where your monitoring stack needs corroborating detections.

Threat narrative

Attacker objective: The attacker’s objective is to move through the environment while avoiding a detection picture that is broad enough to trigger confident triage and containment.

1. Entry occurs when an attacker is first visible through a narrow detection set, often a single alert on initial access or suspicious activity.
2. Escalation becomes harder to spot when the incident depends on one loud detection instead of multiple corroborating signals, increasing the chance of missed privilege misuse or lateral movement.
3. Impact is greater when the organisation cannot distinguish a robust incident from a fragile one, because detection gaps can hide attacker activity until compromise is well established.

NHI Mgmt Group analysis

Detection resilience is now a governance problem, not just a tuning problem. Incident quality measures should tell leaders whether the organisation can still recognise attacker behaviour when one alert source fails. That shifts the conversation from single-rule performance to control redundancy across endpoint, cloud, identity, and network layers. Practitioners should treat detection diversity as a resilience signal, not a cosmetic metric.

Signal strength is the better lens for SOC improvement than alert count. Counting alerts rewards noise and can obscure the fact that one source is carrying the entire incident. A stronger model asks whether multiple detections independently support the same attacker action. The governance value is that teams can prioritise coverage work where the incident picture is most fragile. Practitioners should use this to direct engineering effort toward corroborating controls.

Dominance ratio exposes the hidden fragility of over-centralised telemetry. When one detection contributes most of the evidence, the incident may look rich while remaining structurally weak. That is a useful concept for SOC maturity discussions because it frames monitoring as a dependency problem. Practitioners should map their highest-dominance alert patterns and ask what would vanish if the primary rule went offline.

Detection quality metrics should be linked to framework-based assurance. In NIST CSF terms, this is about improving Detect and Respond functions, and in MITRE ATT&CK terms it is about verifying whether the organisation can observe multiple tactics, not just one noisy stage. For identity-heavy environments, the same logic applies to NHI misuse, where a single alert on secret abuse is rarely enough on its own. Practitioners should use the metric to justify corroborating controls across identity and runtime telemetry.

Fragile incidents are a warning sign of control design debt. If too many incidents sit in the quiet or unstable zones, the issue is not only detection content but also how evidence is distributed across the stack. That makes the named concept here worth tracking: detection resilience debt, the accumulation of monitoring patterns that look adequate until one signal disappears. Practitioners should treat the zone distribution as a roadmap for reducing that debt.

What this signals

Detection resilience debt: teams should treat a concentration of fragile incidents as evidence that coverage is too dependent on a small set of alert sources. The operational answer is not more noise, but better corroboration across identity, endpoint, cloud, and network telemetry. For identity-heavy programmes, review service account and secret activity against the NHI lifecycle guidance in the NHI Lifecycle Management Guide.

A practical next step is to pair this methodology with established monitoring and detection frameworks, especially the NIST Cybersecurity Framework 2.0 and ATT&CK-aligned detection mapping. That gives SOC leaders a cleaner way to show whether detection changes are improving resilience or simply increasing alert volume. The point is programme assurance, not metric theatre.

For practitioners

• Score incidents for corroboration, not just alert totals Add an incident-quality metric that weights unique detections, repeated evidence, and dependence on the loudest source so fragile patterns are easy to spot. Use the resulting score in detection engineering reviews and threat hunt prioritisation.
• Track dominance in your highest-volume detections Identify rules where one telemetry source accounts for most of the incident evidence, then test what disappears when that source is suppressed. Prioritise redundant signals for identity abuse, lateral movement, and cloud escalation paths.
• Separate critical one-alert incidents from weakly supported noise Allow for true early-stage incidents that naturally have fewer signals, but do not let low-volume alerts mask poor coverage elsewhere. Use severity, tactic stage, and corroborating telemetry together before deciding an incident is acceptable.
• Build quarterly trend views for detection resilience Review the share of fragile, risky, decent, and robust incidents each quarter to see whether new detections are reducing dependence on a single control. Link the trend to engineering backlog items and coverage gaps.
• Map identity abuse into incident-quality reviews For service account misuse, token abuse, or API key compromise, verify whether the incident is supported by identity, endpoint, and cloud evidence rather than one alert path. This is especially important where NHI activity can disappear into noisy automation.

Key takeaways

• Incident quality is a resilience measure, not just a triage metric.
• A higher alert count can still hide a fragile detection design if one source carries most of the evidence.
• SOC teams should score corroboration and dominance together to expose where detection coverage still depends on a single control.

Key terms

• Incident Quality: Incident quality is the degree to which an incident is supported by multiple, balanced detections rather than one brittle alert source. It reflects how well the evidence survives loss of a single control and whether the activity is still recognisable enough for confident triage and response.
• Signal Strength: Signal strength is a measure of how much evidence supports an incident after repeated detections are counted in a controlled way. It helps distinguish incidents with real corroboration from those that only look strong because one detection fires many times.
• Dominance Ratio: Dominance ratio measures how much the loudest detection contributes to the total incident evidence. A high ratio means the incident depends heavily on one source, which creates fragility if that source fails, degrades, or becomes too noisy to trust.
• Capped Count: A capped count limits how much repeated firing from one detection contributes to the overall incident score. It preserves evidence value without letting one noisy alert path overwhelm the assessment of diversity and resilience.

What's in the full article

Expel's full article covers the mathematical detail this post intentionally leaves at the analytical level:

• The exact Shannon and Hill Number calculations used to compare incident detection diversity.
• The capped-count method for weighting repeated detections without letting one source dominate the score.
• The dominance-ratio equation and how it separates noisy incidents from robust ones.
• The quarterly stacked-bar approach for tracking whether incident quality is improving over time.

👉 Expel's full article shows the formulas, thresholds, and visual method behind its incident-quality model.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and workload identity. It is designed for practitioners building stronger identity control foundations across modern security programmes.