Most mail between organisations is encrypted only opportunistically, which means an attacker on the network path can strip the STARTTLS offer or present any certificate and the sending server will usually deliver in cleartext anyway. SPF, DKIM and DMARC say who sent a message; they do nothing to keep it confidential in transit. This guide covers the transport layer: how STARTTLS works and fails, the difference between server-to-server relay and client submission, how MTA-STS (RFC 8461) and DANE (RFC 7672) make TLS mandatory and authenticated, how TLS reporting (RFC 8460) shows you what is happening, and how to roll all of it out without losing mail. It is for mail administrators, messaging engineers and security architects running their own MTAs or a cloud mail service.
Key takeaways
- Opportunistic STARTTLS is not protection against an active attacker. RFC 3207 itself notes that removing STARTTLS from the EHLO response stops the client attempting TLS, and most MTAs do not validate certificates on relay.
- MTA-STS lets a receiving domain demand authenticated TLS. A DNS TXT record plus an HTTPS-hosted policy tells senders which MX hosts are valid and that delivery must fail rather than fall back.
- DANE does the same with DNSSEC. TLSA records at
_25._tcp.<mx host>are downgrade resistant without a first-use gap, but require a DNSSEC-signed zone and careful key rollover. - Turn on TLS-RPT before enforcing anything. Daily RFC 8460 reports from senders are the only view you get of failures on other people's servers.
- Submission is a different problem. For clients and applications, RFC 8314 prefers implicit TLS on port 465 and requires TLS 1.2 or later; SMTP AUTH credentials used by devices and apps are non-human identities to manage.
How SMTP transport encryption works
SMTP was designed as a cleartext protocol. RFC 3207 (Standards Track, February 2002) added the STARTTLS extension: the server advertises STARTTLS in its EHLO response, the client issues the command, the server replies 220 Ready to start TLS, and the two negotiate TLS on the same connection. After the handshake both sides must discard what they learned in cleartext and the client sends EHLO again. If the server replies 454 TLS not available, the client decides by local policy whether to continue without TLS, retry or return the mail.
| Flow | Port | Mechanism | Who authenticates whom |
|---|---|---|---|
| MTA to MTA relay | 25 | STARTTLS, opportunistic by default | Usually nobody: certificates often not validated |
| Client or application submission | 465 (submissions) | Implicit TLS from the first byte (RFC 8314) | Client validates server certificate; server authenticates user or app with SMTP AUTH |
| Client or application submission | 587 | STARTTLS upgrade | As above, if TLS is required before AUTH |
| Mail access | 993 (IMAP), 995 (POP3) | Implicit TLS | Client validates server; user authenticates |
RFC 3207 also says a publicly referenced SMTP server must not require STARTTLS to deliver mail locally, because senders that cannot do TLS would then be unable to reach it. That rule is why port 25 relay stays opportunistic unless a policy layer (MTA-STS or DANE) tells senders otherwise for a specific domain. Postfix documentation makes the same point: its server-side smtpd_tls_security_level = encrypt must not be used on a publicly referenced server.
Downgrade and interception risks
- STARTTLS stripping: an on-path attacker deletes
STARTTLSfrom the EHLO response, or rewrites the command, and the sender proceeds in cleartext. RFC 3207 names this attack and recommends that clients and servers be configurable to require TLS for selected hosts. - Certificate substitution: an opportunistic sender that accepts any certificate will happily encrypt to the attacker. Encryption without authentication defeats passive sniffing only.
- MX record manipulation: without DNSSEC or MTA-STS, an attacker who can spoof or poison DNS can point a domain's MX at their own host, which can present a valid certificate for its own name. See the DNS Security Guide.
- Plaintext command injection: CVE-2011-0411, found in Postfix in 2011 and present in many other STARTTLS implementations, let an attacker append commands to the cleartext STARTTLS request that the server then executed inside the TLS session. The fix principle is to discard any input buffered before the switch to TLS. Keep MTAs patched; the Postfix write-up warns that the same class of flaw can recur in other protocols and in proxies that switch from plaintext to TLS.
- Weak protocol versions: MTA-STS requires TLS 1.2 or later for policy-compliant delivery, and RFC 8314 requires mail submission and access servers to support TLS 1.2 or later. See the TLS Server Configuration Baseline Guide.
In ATT&CK terms these map to T1557 Adversary-in-the-Middle and T1040 Network Sniffing, with T1114.002 Remote Email Collection as the objective when mail content or credentials are captured.
MTA-STS: policy-based enforcement over HTTPS
MTA-STS (RFC 8461, Standards Track, September 2018) has two parts. A TXT record at _mta-sts.<domain> signals that a policy exists and carries a version identifier; the policy itself is a text file fetched over HTTPS from https://mta-sts.<domain>/.well-known/mta-sts.txt, which authenticates it through the Web PKI.
| Element | Value or rule (RFC 8461) | Operational note |
|---|---|---|
| TXT record | v=STSv1; id=<1 to 32 letters or digits>;, exactly one valid record | Change id every time the policy changes; a UTC timestamp works well |
| Policy host | HTTPS, 200 response, no redirects followed, certificate valid for mta-sts.<domain> | Media type text/plain; host on any static HTTPS service |
version | STSv1 | Required |
mode | enforce, testing or none | Start with testing |
mx | One line per pattern; full hostname or leftmost-label wildcard such as *.example.net | Must cover every live MX; wildcards can match hosts other tenants control |
max_age | Seconds, maximum 31557600 (about one year) | The RFC suggests weeks or more; use a short value while testing |
In enforce mode a sender must not deliver to a host that fails MX pattern matching, certificate validation (trusted chain, not expired, SAN DNS-ID matching the MX host name) or STARTTLS. The failure is treated as temporary, so mail queues rather than bouncing immediately. In testing mode delivery continues but failures are reported through TLS-RPT. Senders cache the policy for max_age and use the cached copy if a fresh fetch fails.
MTA-STS limits you should know
- First-use gap: protection applies only once a sender has fetched and cached the policy. An attacker who can block DNS on the first lookup can hide the policy. Long
max_agevalues and early refreshes reduce the window. - Web PKI dependency: a mis-issued certificate for the policy host or an MX host can bypass the policy. Watch Certificate Transparency for your
mta-stsand MX names. See the Certificate Transparency Monitoring Guide for Web Teams. - DNS control means policy control: an attacker who can change your DNS can publish a malicious policy with a long
max_agepointing at their MX. Protect registrar and DNS hosting accounts. See the Authoritative DNS and Registrar Account Security Guide. - Delegated subdomains: the RFC warns that services letting untrusted users publish content and TXT records on subdomains could allow policy spoofing.
Cloud mail services
Microsoft documents that Exchange Online always validates MTA-STS on outbound mail and returns NDR codes 5.4.8 (MX host not in policy) or 5.7.5 (certificate validation failed) when an enforcing policy fails. For inbound protection of a domain whose MX points directly to Exchange Online, you publish the TXT record and host the policy yourself, with mx: *.mail.protection.outlook.com; Microsoft notes that this wildcard is specific to its service and that domains with several providers must list each provider's MX hosts.
DANE for SMTP: DNSSEC-based enforcement
DANE for SMTP (RFC 7672, Standards Track, October 2015) publishes TLSA records at _25._tcp.<mx host> in a DNSSEC-signed zone. A sending MTA with a validating resolver that finds secure, usable TLSA records must use TLS and authenticate the server against them, and must not deliver to a server that does not offer STARTTLS. Lookup failures, including bogus DNSSEC results, delay delivery rather than downgrading.
- Recommended record types: RFC 7672 recommends DANE-EE(3) with SPKI(1) and SHA2-256(1) as the primary choice, and DANE-TA(2) with Cert(0) and SHA2-256(1) as the second. PKIX-TA(0) and PKIX-EE(1) are not recommended for SMTP.
- Key rollover: publish the TLSA record for the new key before the certificate changes, so cached records match during the switch. Automated certificate renewal that rotates keys without updating TLSA records is the classic DANE outage.
- Selective STARTTLS conflict: RFC 7672 says servers that offer STARTTLS selectively (for example after greylisting) cannot also publish TLSA records.
- Relationship with MTA-STS: RFC 8461 is designed not to interfere with DANE: a sender must not let an MTA-STS pass override a failed DANE validation. Publishing both is reasonable where you run DNSSEC.
See the DNSSEC Deployment Guide for signing and key management.
Choosing between MTA-STS, DANE and both
| Factor | MTA-STS | DANE |
|---|---|---|
| Prerequisite | HTTPS host and Web PKI certificates for policy host and MX hosts | DNSSEC-signed zone for the MX host names; validating resolvers at senders |
| Trust anchor | Public certificate authorities | DNSSEC chain of trust |
| First-contact protection | No: depends on a cached policy | Yes: every lookup is authenticated |
| Works with cloud MX you do not control | Yes, if the provider's MX names and certificates are stable | Only if the provider publishes TLSA records in a signed zone |
| Main operational risk | Expired policy host certificate, MX list drift | TLSA mismatch after key rotation, DNSSEC signing failures |
| Outbound enforcement | Your MTA must implement MTA-STS validation | Your MTA must validate DNSSEC and TLSA (for example Postfix level dane) |
Decision guidance: a small organisation on a hosted mail service should publish MTA-STS and TLS-RPT for inbound and rely on the provider's outbound validation. An organisation that already runs DNSSEC and its own MTAs should deploy DANE for inbound and enable DANE validation outbound, and add MTA-STS to cover senders that support only MTA-STS. For a sensitive partner flow, configure per-destination mandatory TLS on your MTA (for example Postfix smtp_tls_policy_maps with level secure or verify) regardless of what the partner publishes.
Sender-side settings in practice
Postfix shows the range of client choices through smtp_tls_security_level: none (default), may (opportunistic, falls back to cleartext if the handshake fails), encrypt (mandatory TLS without certificate verification), dane (DANE when usable TLSA records exist, otherwise opportunistic), dane-only, fingerprint, verify (trusted CA and name match) and secure (as verify with DNS-forgery-resistant name checks). On the submission listener, smtpd_tls_auth_only = yes stops AUTH being offered before STARTTLS. RFC 8689 adds the REQUIRETLS SMTP extension, a MAIL FROM parameter by which a sender can demand TLS on every hop, with a TLS-Required: No header for the opposite case; support among receiving servers is limited, so treat it as a targeted control, not a baseline.
Rolling out transport security without losing mail
- Inventory every MX host and inbound path, including gateways, filtering services and backup MX. Confirm each presents a publicly trusted certificate whose SAN matches its MX host name and supports TLS 1.2 or later.
- Publish TLS-RPT first: a TXT record at
_smtp._tls.<domain>withv=TLSRPTv1; rua=mailto:...or anhttps:endpoint. Collect at least two weeks of reports. - Publish MTA-STS in
testingmode with a shortmax_age(for example one day), all MX patterns listed, and a monitored HTTPS host for the policy file. - Fix what the reports show: certificate host mismatches, expired certificates, MX hosts missing from the policy.
- Move to
enforce, change the TXTid, then increasemax_agein steps towards several weeks. - Harden outbound: enable MTA-STS and, where you validate DNSSEC, DANE on your sending MTAs; set mandatory, verified TLS for named high-value partners.
- Harden submission: offer port 465 implicit TLS, require TLS before AUTH on 587, and disable cleartext IMAP and POP3, as RFC 8314 recommends.
- Rollback: to pause enforcement, switch the policy to
testingand change theid. To remove MTA-STS entirely, RFC 8461 says to publishmode: nonewith a shortmax_ageand a newid, wait until every previously served policy has expired, and only then delete the TXT record and policy host. Deleting the record first leaves senders enforcing a stale cached policy.
Detection and measurement
TLS-RPT reports (RFC 8460) are JSON, usually gzip-compressed (application/tlsrpt+gzip), and cover one full UTC day. Each lists the policy type (sts, tlsa or no-policy-found), counts of successful and failed sessions, and failure details.
| TLS-RPT result type | Likely cause | Action |
|---|---|---|
starttls-not-supported | MX not offering STARTTLS, or stripping on the path | Check the MX; if it does offer STARTTLS, investigate the reporting sender's network path |
certificate-host-mismatch | Certificate SAN does not cover the MX host name | Reissue with correct names |
certificate-expired / certificate-not-trusted | Renewal failure or private CA on a public MX | Fix renewal automation; use a public CA |
sts-policy-fetch-error / sts-policy-invalid / sts-webpki-invalid | Policy host down, malformed file or bad certificate on mta-sts host | Monitor the policy URL and its certificate |
tlsa-invalid / dnssec-invalid / dane-required | TLSA mismatch after key change or broken signing | Fix rollover procedure and DNSSEC monitoring |
On your own MTAs, log TLS outcomes per session (Postfix smtp_tls_loglevel = 1 gives a summary line) and alert on cleartext deliveries to domains that publish policies. Also alert on any change to the _mta-sts, _smtp._tls and MX records, and to TLSA records.
Metrics:
- Percentage of inbound and outbound sessions using TLS, and the percentage of those with an authenticated certificate.
- Failed-session rate in TLS-RPT reports per policy type, trending to near zero before and after enforcement.
- Number of owned mail-receiving domains with MTA-STS in
enforce(or DANE) and TLS-RPT published. - Days of remaining validity on MX and
mta-stshost certificates, with alerting well before expiry. - Count of cleartext submission or mail access logins, trending to zero.
Where identity fits
Submission on ports 465 and 587 is authenticated, and much of it is not people: printers, scanners, line-of-business applications, monitoring tools and SaaS integrations hold SMTP AUTH usernames and passwords or API keys for relay services. These credentials are long-lived, often shared and rarely rotated, and when sent over cleartext they can be captured. Inventory them, require TLS before AUTH, scope each to the sender addresses it needs, and rotate them like any other secret. See the SendGrid, Mailgun and Mailchimp API Keys Security Guide and the Exim and Postfix Mail Server Hardening Guide.
Framework mapping
| Framework | Reference | How this guide applies |
|---|---|---|
| NIST SP 800-53 Rev. 5 | SC-8 Transmission Confidentiality and Integrity; SC-8(1) Cryptographic Protection; SC-23 Session Authenticity; SC-20 Secure Name/Address Resolution Service (Authoritative Source); IA-5 Authenticator Management | Mandatory authenticated TLS, DNSSEC for DANE, SMTP AUTH credential management |
| NIST CSF 2.0 | PR.DS-02 Data-in-transit is protected | MTA-STS, DANE and submission TLS as evidence |
| NIST SP 800-177 Rev. 1 | Trustworthy Email | Federal guidance on email authentication and transmission security |
| MITRE ATT&CK | T1557 Adversary-in-the-Middle; T1040 Network Sniffing; T1114.002 Remote Email Collection | Downgrade, interception and the collection goal |
| MITRE CAPEC | CAPEC-94 Adversary in the Middle (AiTM) | STARTTLS stripping as an on-path attack pattern |
Common failure patterns
- Removing MTA-STS by deleting the DNS record: senders keep enforcing the cached policy for up to
max_age, and mail to a migrated MX queues and bounces. Always step down throughmode: none. - Forgetting to change the
id: senders with a cached policy never re-fetch, so a new MX added during a migration fails enforcement. - Policy host certificate expiry: the
mta-stssubdomain is a forgotten static site whose certificate lapses, producingsts-webpki-invalidfailures across all senders. - TLSA records not updated on certificate renewal: automated renewal rotates the key, the DANE-EE record no longer matches, and DANE-validating senders defer all mail.
- Requiring STARTTLS on the public MX: setting server-side mandatory TLS on port 25 rejects legitimate senders and breaks RFC 3207; use MTA-STS or DANE to signal the requirement instead.
- Cleartext AUTH for devices: a multifunction printer allowed to authenticate on port 25 or 587 without TLS leaks a mailbox credential that is often reused elsewhere.
Practitioner checklist
- Confirm every MX presents a publicly trusted certificate that matches its host name and supports TLS 1.2 or later.
- Publish TLS-RPT (
_smtp._tls) for every mail-receiving domain and route reports to a parser. - Publish MTA-STS in
testing, fix failures, thenenforcewith a stepped-upmax_age. - Deploy DANE TLSA records (3 1 1) where your MX zones are DNSSEC-signed, with a rollover procedure tied to certificate renewal.
- Enable MTA-STS and DANE validation on outbound MTAs, and mandatory verified TLS for named sensitive partners.
- Offer implicit TLS on 465, require TLS before AUTH on 587, and disable cleartext IMAP and POP3.
- Inventory and rotate SMTP AUTH credentials and relay API keys held by devices and applications.
- Monitor the
mta-stshost, its certificate, and changes to MX, TXT and TLSA records. - Document the removal procedure (
mode: none, newid, waitmax_age) in your mail migration runbook.
Standards and references
- IETF: RFC 8461 SMTP MTA Strict Transport Security (MTA-STS)
- IETF: RFC 8460 SMTP TLS Reporting
- IETF: RFC 3207 SMTP Service Extension for Secure SMTP over Transport Layer Security
- IETF: RFC 7672 SMTP Security via Opportunistic DANE TLS
- IETF: RFC 8314 Cleartext Considered Obsolete: Use of TLS for Email Submission and Access
- IETF: RFC 8689 SMTP Require TLS Option
- NIST: SP 800-177 Rev. 1 Trustworthy Email
- Microsoft Learn: Enhancing mail flow with MTA-STS
Related NHI Mgmt Group resources: DMARC, SPF and DKIM Guide · Email Security Guide · DNSSEC Deployment Guide · How TLS Works Guide · Exim and Postfix Mail Server Hardening Guide