Why do block counts not prove that a security programme is effective?

Why block counts are a weak effectiveness metric

Block counts tell you volume, not outcome. A programme can block more requests because attack pressure increased, because the control became stricter, or because detection improved. None of those facts proves users are making safer choices or that overall exposure is falling, so the metric needs context before it says anything about effectiveness.

That distinction matters because security programmes are meant to reduce risk, not just increase friction. If you only track stopped attempts, you can miss whether users are still clicking, bypassing controls, reusing weak behaviour, or creating the same exposure through another path.

It is usually more useful to treat block counts as one signal in a control set, not a programme outcome. A good baseline combines blocks with exposure measures, report rates, and user error rates so you can see whether the control is preventing harm or simply intercepting noise.

What block counts can and cannot tell you over time

Block counts are most useful for trend analysis when the environment is stable and the control design has not changed. If the filter logic, policy thresholds, user population, or threat mix changes, the number becomes hard to interpret on its own.

For example, a sudden increase in blocks may indicate a spike in malicious activity, but it may also mean the team tightened rules after a review. A decrease may look positive, yet it could mean attackers shifted tactics, users stopped triggering detection, or logging is incomplete. The metric needs a denominator and a narrative.

If you want a useful operational view, compare blocks with the size of the exposed population, the proportion of users who report suspicious activity, and the frequency of user mistakes that create risk. That gives you a better read on whether the programme is changing behaviour and reducing opportunity, not just counting interceptions.

How to judge effectiveness without over-trusting the counter

Effectiveness should be measured against the intended control objective. If the goal is behaviour change, the evidence should show fewer risky actions, better reporting, and lower repeat error rates. If the goal is exposure reduction, the evidence should show fewer successful paths to harm, not merely more blocked attempts.

That also means separating control performance from threat pressure. A control can look busier in a period of heavy attack and still be working well, or look quieter after a policy change and still be failing. The right question is whether the programme reduced material risk relative to the environment it is defending.

Risk and Threat Considerations

Block-only reporting can hide both false confidence and false alarm. Teams may celebrate a rising block number when the real issue is that the threat environment is worsening, or assume a falling number means improved security when detection has simply become less sensitive. A single count cannot distinguish prevention from volume shifts.

Failure mechanism: The metric collapses different states into one number, so it cannot show whether the control is preventing incidents, displacing them, or missing them entirely.

Impact: Leadership may underinvest in controls that are actually needed, or overrate a programme that has not reduced exposure, behaviour risk, or downstream loss.

Practitioner Guidance

What to verify: Check whether blocks are accompanied by a stable denominator, such as exposed users, events, or requests, and whether the control logic changed during the measurement period. Without that context, trend comparisons are weak.

What to measure: Pair block counts with report rates, repeat user error rates, and exposure reduction. Those measures tell you whether the programme is changing behaviour and shrinking risk, which is the real effectiveness test.

Decision rule: If blocks rise while user errors and exposure also rise, treat the programme as under strain rather than effective. If blocks rise while error rates fall and reporting improves, the control may be working even though the raw count increased.

Practitioner takeaway: Use block counts as a control health signal, not as proof of security value, because effectiveness is shown by reduced risk and changed behaviour, not by interception volume alone.

Related resources from NHI Mgmt Group

• How can organisations prove their pentesting programme is actually effective?
• How should manufacturers prove that security testing under the CRA is effective?
• Which frameworks require financial institutions to prove effective penetration testing and security testing?
• Why do organisations struggle to prove endpoint security controls are effective across every device?