They should test whether all inbound mail is routed through the same inspection path, then compare the same threat set across products and time periods. If Microsoft sits behind an upstream gateway, mailbox-native numbers alone will understate the amount of malicious traffic already stopped before delivery.
How to judge whether Microsoft 365 changes the email protection test
Microsoft 365 does not remove the need to evaluate upstream inspection, it changes what “email protection” means in practice. Teams should determine whether Microsoft is the first security control, the second control, or only the delivery endpoint, because mailbox-native reporting cannot explain traffic stopped earlier in the path. That distinction determines whether product metrics are comparable at all.
The key question is whether the same inbound message is being measured at the same point in the journey. If a gateway, sandbox, or filtering layer sits ahead of Microsoft 365, then one product may be judging raw inbound volume while another is only seeing what survived pre-delivery controls. A fair evaluation needs a common denominator: same mail flow, same threat set, same observation window.
That also means feature comparison should be tied to the actual routing architecture, not the product label. If Microsoft handles only certain mail streams, or if some domains bypass the upstream inspection path, then the apparent detection gap may be an architecture gap rather than a control weakness. Good evaluation work starts by mapping every inbound path and every enforcement point before comparing block rates, verdicts, or incident counts.
Why product numbers can mislead when mail passes through multiple controls
Product dashboards often look precise while still being incomplete. A mailbox-native platform can only report on messages it receives, so upstream rejection, detonation, and quarantine will reduce what appears in downstream telemetry. That can make one vendor look weaker or stronger depending on where in the stack its counters begin and end.
For that reason, comparisons should separate prevention from visibility. Prevention tells you how much malicious traffic was stopped, while visibility tells you how much malicious traffic was observable after prior controls did their work. Comparing those numbers without the same inspection boundary leads to false conclusions about efficacy, tuning, and ROI.
The most reliable test is operational, not promotional: replay or sample the same inbound threat set against each control layer, then compare outcomes by stage. If the upstream gateway already blocks a large share of malicious mail, the downstream Microsoft 365 mailbox view will naturally undercount what the environment actually prevented. That is not a bug in the mailbox product, but it is a bad basis for benchmarking unless the boundary is explicit.
What a fair evaluation should measure instead
A useful evaluation should measure the mail path, the control point, and the outcome together. Teams should look at delivery architecture, pre-delivery filtering, mailbox protections, post-delivery detections, user-reporting signals, and incident response data as one chain. That gives a realistic picture of which layer stopped what, and which layer only confirmed it after the fact.
Security teams should also compare detections over time with the same ruleset, threat mix, and routing assumptions. If the upstream gateway changes, Microsoft policy changes, or message routing shifts between pilot and production, then trend lines lose meaning. For an honest assessment, lock the inspection path before the test and document any exceptions, such as VIP mailboxes, alternate MX records, or direct-to-tenant bypasses.
In practice, the best comparator is not “which product found more bad email,” but “which architecture reduced real exposure most effectively.” That means looking for duplicate coverage, blind spots, latency, and the point at which a message becomes visible to the defender. When Microsoft 365 is part of the stack, the evaluation should explain how it complements upstream inspection, not assume its inbox numbers are the full story.
Risk and Threat Considerations
When mail security controls are layered, the main risk is measurement error that hides a real exposure problem. A downstream-only view can make defenders believe they have low malicious volume when the upstream gateway is simply absorbing most of the attack traffic before Microsoft 365 ever sees it. That weakens tuning, hides exceptions, and can leave bypass paths unreviewed.
Failure mechanism: The same message is counted at different points in the delivery chain, so upstream prevention, downstream quarantine, and mailbox-native detection are treated as if they were directly comparable. That creates misleading efficacy metrics and can mask routing gaps or uneven policy enforcement.
Impact: Teams may underinvest in the control that is actually carrying the load, misjudge product performance, or miss a path that delivers mail outside the expected inspection sequence. In incident response, that can delay root-cause analysis because telemetry does not reflect where the message was first stopped.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the technical controls, while ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | AU-6 — Audit Review, Analysis, and Reporting | Mail-flow comparisons depend on trustworthy stage-specific reporting across controls. |
| CM-8 — System Component Inventory | Accurate email protection evaluation needs a complete inventory of gateways, routes, and enforcement points. | |
| Recommendation — Review mail security logs by control stage so blocked, quarantined, and delivered messages are measured separately. Inventory every inbound mail path and inspection point before comparing product effectiveness. | ||
| NIST CSF 2.0 | DE.CM-01 — Networks and network services are monitored to detect potential cybersecurity events | Email protection evaluation relies on monitoring the mail path to see where malicious traffic is stopped. |
| GV.SC-04 — Cyber supply chain risk management is established and managed | Third-party gateways and Microsoft 365 both form part of the email security supply chain and must be governed as such. | |
| Recommendation — Monitor inbound mail traffic at each control layer to distinguish upstream blocking from downstream detection. Govern all mail-security providers as one dependency chain when validating coverage and handoff points. | ||
| ISO/IEC 27001:2022 | A.8.16 — Monitoring activities | Comparative email protection requires monitoring that preserves where and when mail was stopped. |
| Recommendation — Record the inspection stage for each malicious message so downstream numbers are not misread as total exposure. | ||
Practitioner Guidance
What to verify: Confirm the full inbound mail path for every domain and tenant, including MX records, upstream gateways, transport rules, journaling, and any direct-to-Microsoft routes. If one mailbox set bypasses the gateway, treat its numbers separately instead of blending them into a single scorecard.
What to measure: Track malicious mail at the stage where it was first detected or blocked, then roll those results up by control layer. The useful metrics are stage-specific stop rates, bypass volume, and the share of malicious mail that reached user mailboxes before being contained.
Decision rule: If Microsoft 365 sits behind an upstream gateway, do not use mailbox-native detections alone to judge total email protection effectiveness. Use them as downstream assurance metrics, while the gateway and delivery-path data provide the primary comparison baseline.
Practitioner takeaway: The right question is not whether Microsoft 365 “caught enough email,” but whether the stack is measured from the same control boundary end to end; without that, vendor comparisons will be directionally useful at best and misleading at worst.
Related resources from NHI Mgmt Group
- How should security teams evaluate whether a legacy secure email gateway still adds value in Microsoft 365 or Google Workspace environments?
- How should security teams modernise email protection as collaboration moves deeper into Microsoft 365 and other cloud platforms?
- How should security teams evaluate AI-driven email protection tools?
- How should security teams govern application and device email sent from Microsoft 365?