Email security benchmarks expose the hidden cost of partial visibility

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Proofpoint's blog post: Email security benchmarks are often distorted by where measurement happens, because upstream-blocked threats and post-delivery cleanup are not the same thing.

Why it matters: For IAM and security teams, the lesson is that mailbox protection, message flow architecture, and remediation timing can change what an effectiveness score actually means.

By the numbers:

• The 2026 Verizon DBIR showed the human element was present in 62% of breaches.
• Proofpoint said its median customer running Proofpoint in front of Microsoft Defender catches an additional 27.1% of threats.
• Proofpoint said customers behind Microsoft see a median of 408 advanced threats delivered per 1,000 users.
• Proofpoint reported a median of 41.7 minutes in Microsoft before those threats were condemned by Proofpoint.

Context

Email security measurements often break when they ignore the full mail flow. If a gateway blocks malicious messages before Microsoft 365 or Google Workspace sees them, any benchmark taken inside the mailbox layer will undercount the threats that were already stopped.

That creates a governance problem as much as a tooling problem. Security teams comparing pre-delivery and post-delivery controls need to understand where the measurement happens, because partial visibility can distort procurement decisions, control validation, and claims about residual risk.

Key questions

Q: What breaks when email security measurements only count threats that reached the mailbox layer?

A: You lose the upstream threats that were already blocked, so the score no longer reflects total protection. That creates a false comparison between inline prevention and post-delivery cleanup, and it can make one architecture look weaker simply because it sees a larger threat population. The right test is to compare identical inbound mail streams, not residues.

Q: How should security teams design email protection when attackers move at machine speed across pre-delivery and post-delivery channels?

A: Security teams should treat email security as a continuous control surface, not two disconnected tools. A strong design combines pre-delivery inspection, mailbox-level detection, and shared threat intelligence so novel attacks can be blocked before delivery and contained after delivery. That reduces exposure window, limits operational fragmentation, and improves response when credentials or internal mail flows are abused.

Q: What are the signs that an email benchmark is missing hidden threats?

A: Look for reports that only describe delivered mail, only show remediated messages, or cannot explain where in the mail flow measurement occurred. If the benchmark does not include upstream blocks, inspection position, and raw message data, it may be measuring residue rather than protection.

Q: How should security teams evaluate email protection when Microsoft 365 is part of the stack?

A: They should test whether all inbound mail is routed through the same inspection path, then compare the same threat set across products and time periods. If Microsoft sits behind an upstream gateway, mailbox-native numbers alone will understate the amount of malicious traffic already stopped before delivery.

Technical breakdown

Why mail flow architecture changes benchmark results

Email security benchmarks depend on the inspection point. If a secure email gateway sits in front of Microsoft 365 or Google Workspace, threats blocked there never enter the mailbox provider’s telemetry, so a post-delivery benchmark only sees the residue. That is not a flaw in detection math, it is a measurement boundary problem. The same issue appears when vendors compare inline prevention with API-based or post-delivery cleanup. Those models inspect different populations of mail, so the denominator is not shared. For IAM and mailbox governance, the control question is not who claims the higher score but whether the measurement captures the full inbound threat stream.

Practical implication: Measure controls from the same mail-flow position before comparing efficacy numbers.

Pre-delivery blocking and post-delivery remediation are different controls

Pre-delivery blocking prevents users from ever receiving the message, while post-delivery remediation removes a threat after it has already landed. They solve related but distinct problems. The first reduces exposure and dwell time; the second reduces harm after exposure has occurred. A benchmark that treats them as equivalent can make a cleanup-heavy model look more effective than it is, especially when the user already had time to click, forward, or approve. In practice, the distinction matters most for phishing, BEC, and credential theft, where the attack objective is user interaction. For identity teams, the real risk is not just detection accuracy but the time window in which user trust can be exploited.

Practical implication: Treat post-delivery cleanup as backstop coverage, not as proof of prevention.

Why blind spots matter for identity and AI-assisted attacks

The article highlights threats that may never traverse the detection stack at all, including direct send abuse and tenant-to-tenant delivery bypass. Those paths are especially dangerous because they create blind spots in logging and scoring, which means the organisation cannot govern what it cannot see. That matters for identity programmes because email remains a primary path to credential theft, account takeover, and malicious consent flows. It also matters for AI-assisted environments, where inbox compromise can reach Copilot-like assistants and other trusted workflows. The technical issue is not simply missed malware. It is that control visibility collapses when mail is delivered outside the inspection path.

Practical implication: Route all internal and tenant-to-tenant mail through controls that preserve inspection and evidence.

Threat narrative

Attacker objective: The attacker objective is to reach users with phishing or malware while evading the measurement point that the organisation uses to judge effectiveness.

1. Entry occurs through malicious email delivered by a path that may bypass upstream inspection, including direct send or tenant-to-tenant delivery.
2. Credential theft or malware delivery follows when the message reaches the inbox without being blocked before user exposure.
3. Impact is measured as hidden exposure, because threats that were stopped upstream or remediated later are counted differently from threats prevented outright.

Breaches seen in the wild

• Microsoft verified publisher OAuth phishing 2022: Malicious OAuth apps with a fraudulently obtained Microsoft verified publisher badge tricked UK users into granting mailbox access in 2022.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Measurement point is now a governance control, not just an analytics choice. Email security scores are only meaningful when practitioners know whether they measure upstream prevention, mailbox-layer residue, or post-delivery cleanup. Once different vendors are measuring different populations, the benchmark stops comparing protection and starts comparing architecture. For security leaders, the decision is whether the evaluation captures the entire inbound threat stream or only what survived someone else’s controls.

Partial visibility creates procurement bias. A tool that looks weaker inside the mailbox may actually be intercepting more threats before they ever reach that layer. That means buyers must assess the full mail path, not the console they happen to sit behind. The practitioner conclusion is to validate efficacy using identical threat populations and identical time windows, or to treat the numbers as non-comparable.

Pre-delivery prevention should outrank post-delivery confidence. Cleanup after delivery is useful, but it is not the same as stopping a malicious message before a user can act on it. The governance gap is the assumption that remediation equals prevention. For identity programmes, that assumption is especially dangerous because inbox compromise often becomes the first step in credential theft, consent abuse, or account takeover.

Hidden mail paths now define the blind spots. Direct Send and tenant-to-tenant delivery show how legitimate mail architecture can become an attacker’s bypass route. This is the hidden visibility gap concept the article surfaces: threats are not only missed by a detection engine, they may never enter the engine’s field of view. The practitioner takeaway is to design routing and inspection so that every relevant message is seen before trust is granted.

What this signals

Hidden visibility gap: when security teams measure only delivered mail, they miss the threats that were already stopped upstream and end up comparing different threat populations. That means the evaluation framework, not just the detection engine, becomes part of the control surface.

The practical shift is toward evidence-rich evaluation. Security leaders should demand the raw message record, the inspection point, and the remediation path before they trust any efficacy claim, especially where identity abuse begins with email delivery.

Microsoft 365 environments make this problem more visible because native mail controls, front-line gateways, and API-based add-ons may each see a different slice of the same attack stream. That is why mailbox security governance now has to include path validation, not just policy tuning.

For practitioners

• Measure from the same inspection point Run side-by-side evaluations only when both products inspect the same inbound mail population, with identical time windows and exportable raw data.
• Separate prevention from cleanup Track pre-delivery blocks, post-delivery remediations, and user-exposure dwell time as separate outcomes so the metrics do not collapse into one score.
• Close direct-send bypass paths Review Direct Send, tenant-to-tenant delivery, and other internal routing paths so messages cannot avoid inspection before reaching users.
• Preserve raw evidence for validation Require message IDs, sender, recipient, subject, and timestamps in every evaluation so you can reconcile who caught what and when without vendor-curated summaries.
• Validate controls against credential-theft scenarios Test phishing, BEC, and consent-abuse campaigns separately from generic malware so the evaluation reflects the identity risk that inbox compromise creates.

Key takeaways

• Email security metrics can be misleading when upstream blocks never enter the benchmark view.
• Pre-delivery prevention and post-delivery cleanup answer different operational questions, so they should not be treated as equivalent controls.
• Practitioners need evaluation methods that preserve mail-flow context, raw evidence, and identical threat populations before comparing efficacy claims.

Key terms

• Mail Flow Architecture: The path an email takes from sender to recipient, including gateways, provider filtering, and post-delivery remediation. In security governance, the inspection point matters because different layers see different threat populations, which changes both detection evidence and benchmark interpretation.
• Pre-delivery Blocking: A control that stops malicious email before it reaches the inbox. It reduces user exposure, shortens attacker dwell time, and is usually the stronger prevention outcome when compared with cleanup after delivery.
• Post-delivery remediation: Security action taken after a message has already reached a mailbox or application. It can delete, quarantine, or flag content, but it becomes less effective when the same message has already been copied into downstream systems outside the control boundary.
• Visibility gap: A visibility gap is the point where a security team can no longer reliably see who has access to what, or why that access exists. In identity and data governance, it is a control failure because remediation depends on accurate ownership and current entitlement state.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, human identity, and secrets management from a practitioner perspective. It helps security leaders connect identity controls to real-world operating decisions across modern environments.