Join our Newsletter — 33% off our NHI Course
Home› Guides› Email Gateway and Message Trace Detection Guide: Fields,…
Guide SOC & Detection Engineering

Email Gateway and Message Trace Detection Guide: Fields, Joins and Use Cases

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 14 min read
On this page

Email delivery logs are the only record of what reached, left or bounced off your mail system, yet most detection teams query them by hand during an incident and never build detections on them. Message trace in Exchange Online, Defender for Office 365 email tables, Google Workspace Email Log Search and BigQuery exports, mail transfer agent logs and secure email gateway logs all describe the same thing in different shapes: a message, its envelope, its authentication results, its verdict and where it ended up. This guide is for detection engineers who start from the log they have. It covers what each source contains and its limits, how ATT&CK v20 maps these log sources to techniques, how to join records across sources, the detections worth building, collection and retention, and metrics. Mailbox audit and sign-in logs are covered only where they complete an email detection; see the Microsoft 365 Unified Audit Log Guide for Detection for those. Product limits were checked against Microsoft Learn and Google Workspace documentation on 27 October 2026.

Key takeaways

  • Message trace is a transport log, not content. It tells you sender, recipients, IPs, connectors, status and authentication outcomes. It does not hold the body, and Google's Email Log Search says plainly that you cannot view message contents.
  • Interactive tools have hard limits. Exchange Online message trace searches up to 90 days, returns summary data instantly only for 10 days or less, and the V2 cmdlet is throttled to 100 requests per 5 minutes per tenant. Stream to a SIEM if you need detections.
  • Use the right join key. NetworkMessageId joins Microsoft tables; the RFC 5322 Message-ID (InternetMessageId, rfc2822_message_id) joins across vendors and gateways; MTA queue IDs only join within one server.
  • ATT&CK v20 names the log sources. Analytics in v20 reference m365:messagetrace, Application:Mail and saas:email for techniques such as T1114.003 Email Forwarding Rule, T1684.002 Email Spoofing and T1534 Internal Spearphishing.
  • Delivery is not the end of the story. Post-delivery removal (zero-hour auto purge, admin remediation) and link click events decide whether a missed phish became an incident; detections must use the latest state, not the first verdict.

What each email log source contains

SourceGranularityKey fieldsLimits to know
Exchange Online message trace (EAC, Get-MessageTraceV2, Get-MessageTraceDetailV2)One row per message per recipient; detail events per hopReceived, SenderAddress, RecipientAddress, Subject (first 256 characters), Status, FromIP, ToIP, MessageTraceId; extended reports add network_message_id, original_client_ip, directionality, connector_id, return_path, event_idQuery window 90 days, 10 days per summary query; status can lag 5 to 10 minutes; enhanced and extended reports come from archive, take hours and usually lack the last 24 hours; -ResultSize up to 5,000; 100 requests per 5 minutes
Defender for Office 365 EmailEventsOne row per message per recipientNetworkMessageId, InternetMessageId, SenderFromAddress, SenderMailFromAddress, SenderIPv4, AuthenticationDetails, DeliveryAction, DeliveryLocation, ThreatTypes, DetectionMethods, EmailDirection, IsFirstContact, ForwardingInformation, ExchangeTransportRule, ConnectorsRequires Defender for Office 365; LatestDeliveryAction and LatestDeliveryLocation are not in the Streaming API
EmailPostDeliveryEventsOne row per post-delivery actionActionType (Manual remediation, Phish ZAP, Malware ZAP), ActionTrigger, ActionResult, SourceLocationOnly actions taken after delivery
EmailUrlInfo, EmailAttachmentInfo, UrlClickEventsPer URL, per attachment, per clickURLs and domains, file names and hashes; click ActionType (for example ClickAllowed), IsClickedThrough, UrlChain, Workload (Email, Office, Teams)Clicks from Drafts and Sent Items may lack a usable NetworkMessageId
Google Workspace Email Log Search5 to 7 log entries per recipientSender, Return-Path, recipient, sender and recipient IP, subject, message ID, labels, spam marking, delivery status, post-delivery deletion30 days by default; older messages need recipient and message ID and show post-delivery status only; 1,000 messages and 15,000 entries per search; results in 1 minute to 1 hour
Gmail logs in BigQueryEvent-level rows in the activity tablegmail.event_info.mail_event_type, gmail.message_info.rfc2822_message_id, source.address, destination, connection_info.smtp_tls_state, is_spam, triggered_rule_infoEnterprise, Education Standard and Plus, Frontline and similar editions only; you pay BigQuery storage and query costs
MTA logs (Postfix, Exim, Sendmail)One line per processing step, linked by queue IDClient host and IP, from=, to=, relay=, status= (sent, deferred, bounced), message-id, TLS details, SASL username for authenticated submissionPostfix logs to syslog by default; since 3.4 it can log to a file or stdout via postlogd; file mode defaults to 0600
Secure email gateway logsPer message, with policy and verdictEnvelope, headers, policy matched, verdict, rewritten URLs, quarantine actionsVendor schemas differ; map to a common model before building detections

Two sender fields matter in every source. The envelope sender (MAIL FROM, Return-Path, SenderMailFromAddress) is what SPF checks; the header From (SenderFromAddress) is what users see and what DMARC aligns to. Spoofing and lookalike detections need both. See the DMARC, SPF and DKIM Guide.

How ATT&CK v20 maps email log sources to techniques

ATT&CK v20 detection strategies contain analytics that name concrete log sources and fields, grouped under data components such as Application Log Content. Searching the v20 data for email log sources gives a practical starting map.

TechniqueATT&CK v20 log source in the analyticWhat the analytic looks forField to use
T1114.003 Email Forwarding Rulem365:messagetrace, m365:unifiedX-MS-Exchange-Organization-AutoForwarded on messages; New-InboxRule and Set-InboxRuleEmailEvents.ForwardingInformation, outbound trace to external recipients
T1684.002 Email Spoofingm365:messagetrace, saas:emailSPF, DKIM or DMARC failure, or domain mismatchAuthenticationDetails; envelope versus header domain
T1566.001 Spearphishing Attachmentm365:unified, Application:MailInbound attachments from suspicious or spoofed senders, followed by endpoint executionEmailAttachmentInfo hashes joined to endpoint file events
T1566.002 Spearphishing Linkm365:unified, Application:MailInbound messages with embedded or shortened URLs, then browser or process activityEmailUrlInfo, UrlClickEvents
T1534 Internal SpearphishingApplication:Mail, m365:unifiedInternal-to-internal messages from a compromised account, often after unusual sign-inEmailDirection = Intra-org with URL or attachment verdicts
T1564.008 Email Hiding Rulesm365:unified, ApplicationLog:MailServerNew inbox rules or sieve filters that move or delete messagesAudit log, not trace; trace shows the rule's effect (messages to Deleted Items or RSS folders)
T1137.005 Outlook Rulesm365:messagetraceInbound email matching a crafted rule trigger patternSubject or sender patterns arriving just before endpoint execution

The v20 release also moved impersonation: T1656 Impersonation is revoked, and impersonation and email spoofing now sit under T1684 Social Engineering as T1684.001 Impersonation and T1684.002 Email Spoofing. Update rule tags that still reference T1656.

Joining email records across sources

Most useful email detections need two or more sources. Pick join keys deliberately and normalise early.

KeySet byScopeUse
NetworkMessageIdMicrosoft 365Unique within the tenantJoin EmailEvents, EmailUrlInfo, EmailAttachmentInfo, EmailPostDeliveryEvents, UrlClickEvents and message trace extended data
Internet Message-ID (InternetMessageId, rfc2822_message_id, message-id=)The sending systemGlobal, but the sender controls it and it can be reused or forgedJoin gateway, MTA, Microsoft and Google records for the same message
MTA queue IDEach mail serverOne server, one processing runGroup Postfix or Exim lines into one message record before shipping
Recipient addressEnvelopePer recipient copyCombine with message ID; one message to 50 people is 50 rows
Sender IP and connectorReceiving serverPer hopSeparate direct-to-MX delivery from mail that came through your gateway
  • Deduplicate to latest state: Microsoft documents using summarize arg_max(TimeGenerated, *) by NetworkMessageId, RecipientEmailAddress to keep the latest record per message and recipient in a SIEM. Without this, a message delivered then purged counts twice.
  • Normalise addresses: lower-case, strip display names, and keep domain as a separate field. Keep both envelope and header sender.
  • Normalise verdicts: map vendor verdicts to a small set (clean, spam, bulk, phish, malware) and actions to delivered, junked, quarantined, blocked, removed after delivery.
  • Common schema: mapping to an open schema such as the Elastic Common Schema email.* fields or OCSF email activity lets one rule run across Microsoft, Google and gateway data. See the Detection Engineering Guide.

Detections worth building

Inbound

  • Authentication failure from your own domains: inbound messages where the header From is one of your domains but DMARC fails, delivered to an inbox. These should be zero once DMARC is at p=reject; anything delivered points to an allow-list or a connector bypass.
  • Direct-to-MX bypass: if a gateway sits in front of Microsoft 365 or Google, inbound messages that did not arrive through the gateway's connector or IP range skipped your filtering.
  • First contact with payment language: IsFirstContact true, subject or attachment names about invoices or bank details, sent to finance mailboxes. Useful for BEC triage queues, not paging.
  • Delivered then removed: EmailPostDeliveryEvents with Phish ZAP or Malware ZAP, joined to UrlClickEvents for the same NetworkMessageId before the removal time: those users clicked before the purge and need follow-up.
  • Click-through on a warning: IsClickedThrough true on a URL with a phishing verdict.

Outbound and internal

  • Automatic forwarding to external domains: outbound messages with forwarding information or the AutoForwarded header to a domain the user has never mailed, mapped to T1114.003. Pair with the inbox rule audit event. See the T1114 Email Collection Defensive Guide.
  • Sudden outbound volume from one mailbox: a user sending to hundreds of internal or external recipients in an hour, often with the same subject, is a compromised account spreading phishing (T1534).
  • Outbound blocked as spam or phish: outbound messages your own service quarantined are a strong compromise signal.
  • Non-human senders: application relays, SMTP AUTH service accounts, and email API keys. Baseline which identities send which volumes and alert on new sender identities, new source IPs for an SMTP AUTH account, and spikes. Stolen cloud keys were used to send BEC through Amazon SES in TruffleNet; that mail never touched the corporate mail system, so ingest the delivery service's logs too. See the SendGrid, Mailgun and Mailchimp API Keys Security Guide.
  • Delivery to newly created inbox rules' targets: a burst of inbound messages to Deleted Items, RSS Feeds or Archive for one user, combined with a new rule, suggests a hiding rule (T1564.008).

Transport health that hides attacks

  • Inbound without TLS: Gmail BigQuery exposes smtp_tls_state; MTA logs show TLS per connection. Sudden plaintext delivery from a partner who normally uses TLS can indicate interception or a spoofed path.
  • New or changed connectors and transport rules: trace rows with a connector_id or ExchangeTransportRule you have not seen before. Transport rules that add bypass headers or redirect mail are a persistence method.

Collection, retention and implementation

Interactive message trace is for investigation. Detection needs a continuous feed. Build it in phases.

  1. Inventory mail paths: list every way mail enters and leaves: MX records, gateways, connectors, on-premises relays, application SMTP relays, delivery APIs and marketing platforms. Each is a log source.
  2. Choose the feed: for Microsoft 365, the Defender XDR streaming API or connector into your SIEM for Email* and UrlClickEvents; for Google, BigQuery export; for MTAs, syslog or postlogd file output shipped by an agent; for gateways, the vendor's syslog or API.
  3. Pull message trace only where needed: if you script Get-MessageTraceV2, respect the 100 requests per 5 minutes limit, page with StartingRecipientAddress and the last Received value (V2 has no pagination), and use -MessageTraceId for messages to more than 1,000 recipients. Run it under a dedicated identity with the least role that works, never a Global Administrator; Microsoft documents Organization Management or Exchange Administrator for message trace and recommends the least-privileged role.
  4. Normalise and deduplicate as described above, and test with known messages: send a test phish from an external domain, a spoofed internal domain and an internal account, then confirm each appears once with the right verdict and join.
  5. Write detections in audit mode: run each rule for two to four weeks writing to a review queue, tune allow-lists (newsletters, ticketing systems, partner relays), then promote to alerts. Keep the previous rule version for rollback.
  6. Retention: keep normalised email metadata at least as long as your incident detection window; BEC is often discovered weeks later, past the 30-day Email Log Search default and the 10-day summary trace window. See the SaaS Audit Log Retention and Licensing Guide.

Small team or large SOC

  • Small organisations without Defender for Office 365 or a SIEM can still schedule a daily message trace export of outbound mail to external domains and messages failing DMARC, and review spikes. It is crude but catches forwarding and mass outbound phishing.
  • Large SOCs should stream all email tables, join them with identity and endpoint data, and assign email detections to an owner with tuning reviews. Data volume is high: filter bulk and newsletter mail at ingestion only if you keep a cheaper copy for investigations.

Metrics

  • Coverage: share of mail paths (MX, relays, delivery APIs) feeding the SIEM (target 100 percent).
  • Freshness: median delay from delivery to the record being searchable in the SIEM.
  • Click-before-purge rate: users who clicked a link in a message later removed by ZAP or an admin, per month.
  • Time to remediate: time from first delivery of a phishing campaign to removal from all mailboxes.
  • Precision: share of email alerts closed as true positive, per rule, reviewed monthly.
  • Forwarding exposure: mailboxes forwarding to external domains, trending to zero outside approved exceptions.

Common failure patterns

  • Treating message trace as a SIEM: analysts rely on the 10-day interactive window and discover during a BEC investigation that the relevant mail is older, archived or rate limited.
  • Counting the first verdict: dashboards built on initial DeliveryAction overstate delivered phish (because ZAP removed some) and understate risk (because users clicked first).
  • Ignoring envelope sender: rules on header From alone miss spoofing where SPF passes for an attacker-controlled envelope domain.
  • Missing non-corporate mail paths: application relays, marketing platforms and cloud email APIs send as your domain but log elsewhere.
  • Joining on subject or time: subjects change with forwarding prefixes and time windows mismatch across systems; use message IDs.
  • Over-privileged trace automation: a script with Global Administrator credentials stored on a jump host so it can pull message trace. It becomes a far bigger risk than the detection it supports.

Lessons from incidents

Email logs often provide the only proof of what an attacker sent or received. In the TruffleNet campaign, stolen AWS keys were used to register SES sending identities for hijacked domains and send a fake $50,000 invoice. The fraud was delivered through a cloud email API, not through the victim organisation's mail system, so defenders needed cloud audit logs for identity changes and the delivery service's own sending logs. The lesson is to include every service that can send as you, and the non-human identities that drive them, in the email detection inventory. For response steps when a mailbox or sender is compromised, see the Business Email Compromise Response Playbook.

Framework mapping

Control areaNIST SP 800-53 Rev 5NIST CSF 2.0CIS Controls v8.1ISO/IEC 27001:2022 Annex A
Email log collection and retentionAU-2, AU-11, AU-12DE.CM-098.2, 8.10A.8.15
Analysis and correlationAU-6, SI-4DE.AE-02, DE.AE-038.11, 13.1A.8.16
Email protections producing the verdictsSI-3, SI-8DE.CM-019.5, 9.6, 9.7A.8.7, A.8.23
Least privilege for trace automationAC-6PR.AA-055.4A.8.2

Practitioner checklist

  • Inventory every mail path, including relays, gateways and cloud email APIs, and confirm each logs to the SIEM.
  • Stream Defender email tables or Gmail BigQuery logs instead of relying on interactive message trace.
  • Join on NetworkMessageId within Microsoft 365 and on Internet Message-ID across vendors; deduplicate to the latest state per recipient.
  • Keep envelope sender and header From as separate normalised fields.
  • Tag rules with ATT&CK v20 IDs and replace any T1656 tags with T1684.001 or T1684.002.
  • Build detections for external auto-forwarding, DMARC failures from your own domains, direct-to-MX bypass and click-before-purge.
  • Baseline non-human senders (SMTP AUTH accounts, relays, API keys) and alert on new identities and source IPs.
  • Run new rules in audit mode for two to four weeks before alerting, and record tuning decisions.
  • Run any message trace automation under a least-privileged dedicated identity within the throttling limits.
  • Retain normalised email metadata beyond the default tool windows to cover late-discovered BEC.

Standards and references

Related NHI Mgmt Group resources: Email Security Guide · Email Forensics and Header Analysis Guide · KQL for Threat Hunting Guide · Exchange Online Security Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.