An email security gateway that blocked well in its first month can lose its edge without a single alert: a new domain routes around it, allow-list entries outlive their reason, quarantine releases creep up and phishing that slips through takes hours to pull back. This health check gives the gateway owner a fixed set of quarterly measures, each with a formula, a data source and green, amber and red thresholds, plus a findings log and a map from each finding to the guide that fixes it. It applies to any gateway model: an MX-based secure email gateway, an API-based integrated cloud email security service, or a mix of both. It is written for messaging and email security owners, security operations, security risk and internal audit.
What this health check covers
The check covers inbound mail flow through the gateway, the filtering and detonation it performs, quarantine handling, post-delivery remediation, the handling of user-reported phishing, transport security reporting and the gateway's own administration and policy integrity. For how gateway models differ and how sender authentication works, see the Email Security Guide. For how to lock down the gateway's admin and quarantine roles, signing keys and connector identities, see Securing the email security gateway itself. This page does not repeat that hardening; it measures whether it is still in place and whether the gateway is still catching what it should.
Think of it as an email security audit checklist with numbers attached. The overview guide tells you what good looks like; this health check asks, every quarter, how far your live gateway has drifted from it. The results support continuous monitoring evidence under NIST SP 800-53 CA-7 and the gateway-specific controls SI-3 Malicious Code Protection and SI-8 Spam Protection.
Key takeaways
- Bypass is the first thing to measure. Mail that reaches mailboxes without passing the gateway, through a forgotten MX record, a direct connector or a newly added domain, is not filtered at all, so mail flow coverage comes before detection metrics.
- Allow-lists are where gateways decay. Every sender or domain bypass entry is a standing exception; measure expiry, ownership and how often allow-listed mail is later removed as malicious.
- Measure what got through, not only what was blocked. Block counts rise with spam volume and say little; post-delivery removals per thousand messages and time to remove them show real residual risk.
- Quarantine release is a control. Releases of malware or high-confidence phishing should be rare, approved and traceable, and a rising rate of false positive releases signals tuning debt.
- Thresholds are a starting point. The bands below are NHI Mgmt Group recommendations; record your own in the email security standard and keep them stable so trends are comparable.
How to use this template
Copy the metrics table and the findings log into your own tracker, then run this Email security gateway health check at the cadence set out below. Collect each figure from the data sources described, score it against the green, amber and red thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review so you can compare results quarter on quarter: the trend matters as much as any single score.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| Operational dashboard (EG-01, EG-05, EG-08) | Monthly | Email security gateway owner | Security operations lead | Dated exports of mail flow, remediation times and quarantine releases |
| Quarterly health check (all metrics) | Quarterly | Email security gateway owner | Head of security operations or security risk | Scorecard, raw exports, findings log update, sign-off record |
| Annual review | Yearly | Gateway owner with internal audit | CISO | Sample of 20 released messages and 20 allow-list entries traced to approval, threshold review, admin role recertification |
| Event-driven check | After a domain acquisition, mail platform change, gateway migration or a phishing incident that reached users | Messaging team | Head of security operations | Mail flow and MX reconciliation, test message results |
Fix one measurement date per quarter and pull every export within one working day of it. Figures on quarantine releases and allow-list changes should be signed off by someone who cannot make those changes, in line with separation of duties (NIST SP 800-53 AC-5).
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Malicious" means malware or phishing confirmed by the gateway verdict, a security analyst or a sandbox result.
| ID | Metric | Formula (numerator / denominator) | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| EG-01 | Mail flow coverage | Inbound external messages scanned by the gateway / inbound external messages received by the mail platform | Gateway message trace; mail platform message trace | 99.5% or more | 98% to below 99.5% | Below 98%, or any route that bypasses scanning |
| EG-02 | Domain routing coverage | Accepted mail domains whose MX or connector route sends inbound mail through the gateway / accepted domains on the mail platform | DNS MX lookups; mail platform accepted domain list | 100% | Gap on a non-receiving domain with a plan | Any receiving domain not routed |
| EG-03 | Allow-list hygiene | Sender, domain and IP allow or bypass entries past expiry, with no expiry or with no owner / all such entries | Gateway policy export | 0% | Above 0% to 5% | Above 5%, or any entry allowing a whole public mail provider domain |
| EG-04 | Allow-listed malicious rate | Messages delivered by an allow or bypass rule and later classified malicious / messages delivered by allow or bypass rules | Gateway message trace with rule hit; remediation log | 0 | Any, each investigated | Any repeat from the same entry |
| EG-05 | Post-delivery removal time | Median minutes from the first malicious verdict or confirmed user report to removal from all mailboxes, for campaigns in the quarter | Remediation or clawback log; case records | 30 minutes or less | Above 30 minutes to 120 minutes | More than 120 minutes |
| EG-06 | Missed threat rate | Malicious messages found after delivery (removed by clawback or confirmed from user reports) per 1,000 inbound messages delivered | Remediation log; gateway delivery statistics | Falling and below your agreed baseline | Flat | Rising for two quarters |
| EG-07 | User report triage time | Median minutes from a user reporting a message to an analyst or automated verdict | Report mailbox or reporting add-in log; case records | 60 minutes or less in working hours | Over 60 to 240 minutes | More than 240 minutes, or reports unread |
| EG-08 | High-risk quarantine releases | Releases of messages with a malware or high-confidence phishing verdict / such messages quarantined; plus the share of those releases with a recorded approver and case reference | Quarantine audit log | Below 0.1%, all with approver | 0.1% to 0.5%, all with approver | Above 0.5%, or any release without approver |
| EG-09 | False positive release rate | Quarantined messages released as legitimate on request / messages quarantined as spam or bulk | Quarantine audit log; release request records | Below 2% | 2% to 5% | Above 5% |
| EG-10 | Inbound DMARC policy honoured | Inbound messages failing DMARC from domains publishing p=reject or p=quarantine that were rejected or quarantined / all such failing messages | Gateway authentication results in message trace | 99% or more | 95% to below 99% | Below 95%, or failing messages delivered by an override rule |
| EG-11 | Detonation coverage | Inbound attachments and links of in-scope risky types sent for sandbox analysis or time-of-click checking / inbound attachments and links of those types | Gateway sandbox and URL analysis logs | 98% or more | 90% to below 98% | Below 90%, or sandbox timeouts delivering unscanned files |
| EG-12 | Outbound transport failure rate | Failed TLS sessions / total sessions in SMTP TLS reports received for your sending domains | TLS-RPT aggregate reports | Below 0.5% | 0.5% to 2% | Above 2%, or reports not collected |
| EG-13 | Policy and admin drift | Gateway policy, rule, connector and admin role changes without an approved change reference / all such changes in the quarter | Gateway audit log; change records | 0 | Low-risk changes recorded late | Any unapproved bypass rule, connector or admin |
Report each metric with its numerator and denominator. A single red bypass finding (an unrouted receiving domain, an unapproved connector) outweighs any green detection figure.
Notes on the harder metrics
- EG-01 mail flow coverage: for an MX-based gateway, compare messages accepted by the mail platform from the internet with messages the gateway passed on. For an API-based service, compare mailboxes in scope with mailboxes the service monitors, and messages delivered with messages it evaluated. Direct-to-platform delivery that ignores MX records is the usual gap.
- EG-04 allow-listed malicious rate: this is the clearest sign that an allow-list entry has outlived its reason or that a trusted sender's account has been taken over. Trace each hit to the entry and its owner.
- EG-06 missed threat rate: it depends on how many reports users make, so read it alongside the report rate. A falling missed rate with a falling report rate may mean users stopped reporting.
- EG-10 DMARC: RFC 9989 (May 2026) is the current DMARC specification and obsoletes RFC 7489. Measure whether the gateway acts on the sender's published policy, and log any local override. See the DMARCbis Changes Guide.
- EG-12 transport: TLS reporting is defined in RFC 8460; aggregate reports come from receiving domains, so this metric covers your outbound mail as others see it. Your own domains' DMARC enforcement belongs to the DMARC, SPF and DKIM Guide.
How to collect the data
- Mail platform accepted domain list and DNS: list accepted domains and resolve their MX records on the measurement date for EG-02.
- Message traces from both sides: export inbound message counts from the gateway and from the mail platform for the same period for EG-01, and the gateway's per-message verdict, rule hit and authentication results for EG-04 and EG-10.
- Gateway policy export: pull every allow, bypass and block entry with owner, reason, created date and expiry for EG-03.
- Remediation and clawback log: export each post-delivery removal with first verdict time, removal completion time and mailbox count for EG-05 and EG-06.
- User report records: pull report timestamps and verdict timestamps from the reporting mailbox or case system for EG-07.
- Quarantine audit log: export releases with original verdict, releaser, approver and case reference for EG-08 and EG-09.
- Sandbox and URL analysis logs: count risky attachment and link types received, sent for analysis, and timed out for EG-11.
- TLS-RPT aggregate reports: collect reports sent to the address in your TLS reporting DNS record and sum sessions by result for EG-12.
- Gateway audit log and change records: list policy, connector and admin role changes and match each to a change reference for EG-13.
For every red metric, test before reporting: send a harmless test message to the affected domain or route and confirm in the gateway trace whether it was scanned. Keep dated exports with the findings log.
Red flags that need action now
- A receiving domain whose MX record points straight at the mail platform rather than the gateway, or a connector that accepts internet mail without scanning.
- An allow-list entry covering an entire public webmail provider, or any entry that delivered confirmed malicious mail this quarter.
- A malware or high-confidence phishing release with no approver or case reference.
- Median post-delivery removal time above two hours for a campaign that reached many mailboxes.
- User reports sitting unread for more than a working day.
- Sandbox timeouts configured to deliver attachments unscanned.
- A new transport rule, connector or admin account with no change record, which can also be a sign of attacker persistence. See If your email security gateway is compromised.
- Gateway audit logs not reaching the SIEM, or a gap in forwarding during the quarter.
Findings log
Keep one running log across quarters. Escalate anything red for two consecutive quarters to the CISO, and anything repeated three times to the risk register, tracked as a plan of action and milestones (NIST SP 800-53 CA-5).
| ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-EMAIL-01 | [Domain] receives mail directly, bypassing the gateway | EG-02 | Red | [Messaging team] | [Date] | Open |
| HC-EMAIL-02 | [n] allow-list entries with no expiry or owner | EG-03 | Amber | [Gateway owner] | [Date] | Open |
| HC-EMAIL-03 | [n] high-confidence phishing releases without approver | EG-08 | Red | [Security operations lead] | [Date] | Open |
| HC-EMAIL-04 | Median removal time [n] minutes for [campaign] | EG-05 | Amber | [Security operations lead] | [Date] | Open |
| HC-EMAIL-05 | TLS-RPT reports not collected for [domain] | EG-12 | Amber | [Messaging team] | [Date] | Open |
Remediation map
| Finding type | Where to fix it |
|---|---|
| Bypass routes, unrouted domains (EG-01, EG-02) | Email Security Guide: MX routing, connectors and bypass |
| Allow-list sprawl, allow-listed malicious mail (EG-03, EG-04) | Securing the email security gateway itself: policy and allow-list integrity |
| Unapproved releases, release rights too wide (EG-08) | Securing the email security gateway itself: admin and quarantine roles |
| Unapproved rules, connectors or admins (EG-13) | Securing the email security gateway itself: service accounts, API applications and connectors, and the gateway's own logs |
| Slow removal, slow triage (EG-05, EG-07) | Email security gateway day-2 operations runbook |
| High false positive releases, missed threats (EG-06, EG-09, EG-11) | Email Security Guide: filtering, sandboxing and post-delivery remediation |
| DMARC not honoured, transport failures (EG-10, EG-12) | DMARC, SPF and DKIM Failure Troubleshooting Guide |
| Missed threats that reached finance users | Business Email Compromise Response Playbook |
Related guides
- Securing the email security gateway itself: the hardening this check measures against.
- Email Security Guide: gateway models, sender authentication and filtering.
- Email Security Maturity Model: where the scorecard results place you.
- Security Operations Metrics Guide: the wider SOC metrics that triage times feed.
- If your email security gateway is compromised: response steps when a red flag turns into an incident.
- Email security gateway day-2 operations runbook: routine tasks that keep these metrics green.
References
- NIST: SP 800-177 Rev. 1 Trustworthy Email
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls (SI-3, SI-8, SC-8, CA-5, CA-7, AC-5)
- IETF: RFC 9989 Domain-Based Message Authentication, Reporting, and Conformance (DMARC)
- IETF: RFC 8460 SMTP TLS Reporting
- NIST: SP 800-55 Vol. 1 Measurement Guide for Information Security
- CIS: Control 9 Email and Web Browser Protections
- MITRE ATT&CK: T1566 Phishing
Related NHI Mgmt Group resources: Securing the email security gateway itself, Email Security Guide, Email Security Maturity Model, Security Operations Metrics Guide, If your email security gateway is compromised, Email security gateway day-2 operations runbook.