Join our Newsletter — 33% off our NHI Course
Home› Guides› SMTP Security, STARTTLS and MTA-STS Guide: Stop Downgrades…
Guide Email & Communications Security

SMTP Security, STARTTLS and MTA-STS Guide: Stop Downgrades and Enforce Mail Transport Encryption

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 15 min read
On this page

Most mail between organisations is encrypted only opportunistically, which means an attacker on the network path can strip the STARTTLS offer or present any certificate and the sending server will usually deliver in cleartext anyway. SPF, DKIM and DMARC say who sent a message; they do nothing to keep it confidential in transit. This guide covers the transport layer: how STARTTLS works and fails, the difference between server-to-server relay and client submission, how MTA-STS (RFC 8461) and DANE (RFC 7672) make TLS mandatory and authenticated, how TLS reporting (RFC 8460) shows you what is happening, and how to roll all of it out without losing mail. It is for mail administrators, messaging engineers and security architects running their own MTAs or a cloud mail service.

Key takeaways

  • Opportunistic STARTTLS is not protection against an active attacker. RFC 3207 itself notes that removing STARTTLS from the EHLO response stops the client attempting TLS, and most MTAs do not validate certificates on relay.
  • MTA-STS lets a receiving domain demand authenticated TLS. A DNS TXT record plus an HTTPS-hosted policy tells senders which MX hosts are valid and that delivery must fail rather than fall back.
  • DANE does the same with DNSSEC. TLSA records at _25._tcp.<mx host> are downgrade resistant without a first-use gap, but require a DNSSEC-signed zone and careful key rollover.
  • Turn on TLS-RPT before enforcing anything. Daily RFC 8460 reports from senders are the only view you get of failures on other people's servers.
  • Submission is a different problem. For clients and applications, RFC 8314 prefers implicit TLS on port 465 and requires TLS 1.2 or later; SMTP AUTH credentials used by devices and apps are non-human identities to manage.

How SMTP transport encryption works

SMTP was designed as a cleartext protocol. RFC 3207 (Standards Track, February 2002) added the STARTTLS extension: the server advertises STARTTLS in its EHLO response, the client issues the command, the server replies 220 Ready to start TLS, and the two negotiate TLS on the same connection. After the handshake both sides must discard what they learned in cleartext and the client sends EHLO again. If the server replies 454 TLS not available, the client decides by local policy whether to continue without TLS, retry or return the mail.

FlowPortMechanismWho authenticates whom
MTA to MTA relay25STARTTLS, opportunistic by defaultUsually nobody: certificates often not validated
Client or application submission465 (submissions)Implicit TLS from the first byte (RFC 8314)Client validates server certificate; server authenticates user or app with SMTP AUTH
Client or application submission587STARTTLS upgradeAs above, if TLS is required before AUTH
Mail access993 (IMAP), 995 (POP3)Implicit TLSClient validates server; user authenticates

RFC 3207 also says a publicly referenced SMTP server must not require STARTTLS to deliver mail locally, because senders that cannot do TLS would then be unable to reach it. That rule is why port 25 relay stays opportunistic unless a policy layer (MTA-STS or DANE) tells senders otherwise for a specific domain. Postfix documentation makes the same point: its server-side smtpd_tls_security_level = encrypt must not be used on a publicly referenced server.

Downgrade and interception risks

  • STARTTLS stripping: an on-path attacker deletes STARTTLS from the EHLO response, or rewrites the command, and the sender proceeds in cleartext. RFC 3207 names this attack and recommends that clients and servers be configurable to require TLS for selected hosts.
  • Certificate substitution: an opportunistic sender that accepts any certificate will happily encrypt to the attacker. Encryption without authentication defeats passive sniffing only.
  • MX record manipulation: without DNSSEC or MTA-STS, an attacker who can spoof or poison DNS can point a domain's MX at their own host, which can present a valid certificate for its own name. See the DNS Security Guide.
  • Plaintext command injection: CVE-2011-0411, found in Postfix in 2011 and present in many other STARTTLS implementations, let an attacker append commands to the cleartext STARTTLS request that the server then executed inside the TLS session. The fix principle is to discard any input buffered before the switch to TLS. Keep MTAs patched; the Postfix write-up warns that the same class of flaw can recur in other protocols and in proxies that switch from plaintext to TLS.
  • Weak protocol versions: MTA-STS requires TLS 1.2 or later for policy-compliant delivery, and RFC 8314 requires mail submission and access servers to support TLS 1.2 or later. See the TLS Server Configuration Baseline Guide.

In ATT&CK terms these map to T1557 Adversary-in-the-Middle and T1040 Network Sniffing, with T1114.002 Remote Email Collection as the objective when mail content or credentials are captured.

MTA-STS: policy-based enforcement over HTTPS

MTA-STS (RFC 8461, Standards Track, September 2018) has two parts. A TXT record at _mta-sts.<domain> signals that a policy exists and carries a version identifier; the policy itself is a text file fetched over HTTPS from https://mta-sts.<domain>/.well-known/mta-sts.txt, which authenticates it through the Web PKI.

ElementValue or rule (RFC 8461)Operational note
TXT recordv=STSv1; id=<1 to 32 letters or digits>;, exactly one valid recordChange id every time the policy changes; a UTC timestamp works well
Policy hostHTTPS, 200 response, no redirects followed, certificate valid for mta-sts.<domain>Media type text/plain; host on any static HTTPS service
versionSTSv1Required
modeenforce, testing or noneStart with testing
mxOne line per pattern; full hostname or leftmost-label wildcard such as *.example.netMust cover every live MX; wildcards can match hosts other tenants control
max_ageSeconds, maximum 31557600 (about one year)The RFC suggests weeks or more; use a short value while testing

In enforce mode a sender must not deliver to a host that fails MX pattern matching, certificate validation (trusted chain, not expired, SAN DNS-ID matching the MX host name) or STARTTLS. The failure is treated as temporary, so mail queues rather than bouncing immediately. In testing mode delivery continues but failures are reported through TLS-RPT. Senders cache the policy for max_age and use the cached copy if a fresh fetch fails.

MTA-STS limits you should know

  • First-use gap: protection applies only once a sender has fetched and cached the policy. An attacker who can block DNS on the first lookup can hide the policy. Long max_age values and early refreshes reduce the window.
  • Web PKI dependency: a mis-issued certificate for the policy host or an MX host can bypass the policy. Watch Certificate Transparency for your mta-sts and MX names. See the Certificate Transparency Monitoring Guide for Web Teams.
  • DNS control means policy control: an attacker who can change your DNS can publish a malicious policy with a long max_age pointing at their MX. Protect registrar and DNS hosting accounts. See the Authoritative DNS and Registrar Account Security Guide.
  • Delegated subdomains: the RFC warns that services letting untrusted users publish content and TXT records on subdomains could allow policy spoofing.

Cloud mail services

Microsoft documents that Exchange Online always validates MTA-STS on outbound mail and returns NDR codes 5.4.8 (MX host not in policy) or 5.7.5 (certificate validation failed) when an enforcing policy fails. For inbound protection of a domain whose MX points directly to Exchange Online, you publish the TXT record and host the policy yourself, with mx: *.mail.protection.outlook.com; Microsoft notes that this wildcard is specific to its service and that domains with several providers must list each provider's MX hosts.

DANE for SMTP: DNSSEC-based enforcement

DANE for SMTP (RFC 7672, Standards Track, October 2015) publishes TLSA records at _25._tcp.<mx host> in a DNSSEC-signed zone. A sending MTA with a validating resolver that finds secure, usable TLSA records must use TLS and authenticate the server against them, and must not deliver to a server that does not offer STARTTLS. Lookup failures, including bogus DNSSEC results, delay delivery rather than downgrading.

  • Recommended record types: RFC 7672 recommends DANE-EE(3) with SPKI(1) and SHA2-256(1) as the primary choice, and DANE-TA(2) with Cert(0) and SHA2-256(1) as the second. PKIX-TA(0) and PKIX-EE(1) are not recommended for SMTP.
  • Key rollover: publish the TLSA record for the new key before the certificate changes, so cached records match during the switch. Automated certificate renewal that rotates keys without updating TLSA records is the classic DANE outage.
  • Selective STARTTLS conflict: RFC 7672 says servers that offer STARTTLS selectively (for example after greylisting) cannot also publish TLSA records.
  • Relationship with MTA-STS: RFC 8461 is designed not to interfere with DANE: a sender must not let an MTA-STS pass override a failed DANE validation. Publishing both is reasonable where you run DNSSEC.

See the DNSSEC Deployment Guide for signing and key management.

Choosing between MTA-STS, DANE and both

FactorMTA-STSDANE
PrerequisiteHTTPS host and Web PKI certificates for policy host and MX hostsDNSSEC-signed zone for the MX host names; validating resolvers at senders
Trust anchorPublic certificate authoritiesDNSSEC chain of trust
First-contact protectionNo: depends on a cached policyYes: every lookup is authenticated
Works with cloud MX you do not controlYes, if the provider's MX names and certificates are stableOnly if the provider publishes TLSA records in a signed zone
Main operational riskExpired policy host certificate, MX list driftTLSA mismatch after key rotation, DNSSEC signing failures
Outbound enforcementYour MTA must implement MTA-STS validationYour MTA must validate DNSSEC and TLSA (for example Postfix level dane)

Decision guidance: a small organisation on a hosted mail service should publish MTA-STS and TLS-RPT for inbound and rely on the provider's outbound validation. An organisation that already runs DNSSEC and its own MTAs should deploy DANE for inbound and enable DANE validation outbound, and add MTA-STS to cover senders that support only MTA-STS. For a sensitive partner flow, configure per-destination mandatory TLS on your MTA (for example Postfix smtp_tls_policy_maps with level secure or verify) regardless of what the partner publishes.

Sender-side settings in practice

Postfix shows the range of client choices through smtp_tls_security_level: none (default), may (opportunistic, falls back to cleartext if the handshake fails), encrypt (mandatory TLS without certificate verification), dane (DANE when usable TLSA records exist, otherwise opportunistic), dane-only, fingerprint, verify (trusted CA and name match) and secure (as verify with DNS-forgery-resistant name checks). On the submission listener, smtpd_tls_auth_only = yes stops AUTH being offered before STARTTLS. RFC 8689 adds the REQUIRETLS SMTP extension, a MAIL FROM parameter by which a sender can demand TLS on every hop, with a TLS-Required: No header for the opposite case; support among receiving servers is limited, so treat it as a targeted control, not a baseline.

Rolling out transport security without losing mail

  1. Inventory every MX host and inbound path, including gateways, filtering services and backup MX. Confirm each presents a publicly trusted certificate whose SAN matches its MX host name and supports TLS 1.2 or later.
  2. Publish TLS-RPT first: a TXT record at _smtp._tls.<domain> with v=TLSRPTv1; rua=mailto:... or an https: endpoint. Collect at least two weeks of reports.
  3. Publish MTA-STS in testing mode with a short max_age (for example one day), all MX patterns listed, and a monitored HTTPS host for the policy file.
  4. Fix what the reports show: certificate host mismatches, expired certificates, MX hosts missing from the policy.
  5. Move to enforce, change the TXT id, then increase max_age in steps towards several weeks.
  6. Harden outbound: enable MTA-STS and, where you validate DNSSEC, DANE on your sending MTAs; set mandatory, verified TLS for named high-value partners.
  7. Harden submission: offer port 465 implicit TLS, require TLS before AUTH on 587, and disable cleartext IMAP and POP3, as RFC 8314 recommends.
  8. Rollback: to pause enforcement, switch the policy to testing and change the id. To remove MTA-STS entirely, RFC 8461 says to publish mode: none with a short max_age and a new id, wait until every previously served policy has expired, and only then delete the TXT record and policy host. Deleting the record first leaves senders enforcing a stale cached policy.

Detection and measurement

TLS-RPT reports (RFC 8460) are JSON, usually gzip-compressed (application/tlsrpt+gzip), and cover one full UTC day. Each lists the policy type (sts, tlsa or no-policy-found), counts of successful and failed sessions, and failure details.

TLS-RPT result typeLikely causeAction
starttls-not-supportedMX not offering STARTTLS, or stripping on the pathCheck the MX; if it does offer STARTTLS, investigate the reporting sender's network path
certificate-host-mismatchCertificate SAN does not cover the MX host nameReissue with correct names
certificate-expired / certificate-not-trustedRenewal failure or private CA on a public MXFix renewal automation; use a public CA
sts-policy-fetch-error / sts-policy-invalid / sts-webpki-invalidPolicy host down, malformed file or bad certificate on mta-sts hostMonitor the policy URL and its certificate
tlsa-invalid / dnssec-invalid / dane-requiredTLSA mismatch after key change or broken signingFix rollover procedure and DNSSEC monitoring

On your own MTAs, log TLS outcomes per session (Postfix smtp_tls_loglevel = 1 gives a summary line) and alert on cleartext deliveries to domains that publish policies. Also alert on any change to the _mta-sts, _smtp._tls and MX records, and to TLSA records.

Metrics:

  • Percentage of inbound and outbound sessions using TLS, and the percentage of those with an authenticated certificate.
  • Failed-session rate in TLS-RPT reports per policy type, trending to near zero before and after enforcement.
  • Number of owned mail-receiving domains with MTA-STS in enforce (or DANE) and TLS-RPT published.
  • Days of remaining validity on MX and mta-sts host certificates, with alerting well before expiry.
  • Count of cleartext submission or mail access logins, trending to zero.

Where identity fits

Submission on ports 465 and 587 is authenticated, and much of it is not people: printers, scanners, line-of-business applications, monitoring tools and SaaS integrations hold SMTP AUTH usernames and passwords or API keys for relay services. These credentials are long-lived, often shared and rarely rotated, and when sent over cleartext they can be captured. Inventory them, require TLS before AUTH, scope each to the sender addresses it needs, and rotate them like any other secret. See the SendGrid, Mailgun and Mailchimp API Keys Security Guide and the Exim and Postfix Mail Server Hardening Guide.

Framework mapping

FrameworkReferenceHow this guide applies
NIST SP 800-53 Rev. 5SC-8 Transmission Confidentiality and Integrity; SC-8(1) Cryptographic Protection; SC-23 Session Authenticity; SC-20 Secure Name/Address Resolution Service (Authoritative Source); IA-5 Authenticator ManagementMandatory authenticated TLS, DNSSEC for DANE, SMTP AUTH credential management
NIST CSF 2.0PR.DS-02 Data-in-transit is protectedMTA-STS, DANE and submission TLS as evidence
NIST SP 800-177 Rev. 1Trustworthy EmailFederal guidance on email authentication and transmission security
MITRE ATT&CKT1557 Adversary-in-the-Middle; T1040 Network Sniffing; T1114.002 Remote Email CollectionDowngrade, interception and the collection goal
MITRE CAPECCAPEC-94 Adversary in the Middle (AiTM)STARTTLS stripping as an on-path attack pattern

Common failure patterns

  • Removing MTA-STS by deleting the DNS record: senders keep enforcing the cached policy for up to max_age, and mail to a migrated MX queues and bounces. Always step down through mode: none.
  • Forgetting to change the id: senders with a cached policy never re-fetch, so a new MX added during a migration fails enforcement.
  • Policy host certificate expiry: the mta-sts subdomain is a forgotten static site whose certificate lapses, producing sts-webpki-invalid failures across all senders.
  • TLSA records not updated on certificate renewal: automated renewal rotates the key, the DANE-EE record no longer matches, and DANE-validating senders defer all mail.
  • Requiring STARTTLS on the public MX: setting server-side mandatory TLS on port 25 rejects legitimate senders and breaks RFC 3207; use MTA-STS or DANE to signal the requirement instead.
  • Cleartext AUTH for devices: a multifunction printer allowed to authenticate on port 25 or 587 without TLS leaks a mailbox credential that is often reused elsewhere.

Practitioner checklist

  • Confirm every MX presents a publicly trusted certificate that matches its host name and supports TLS 1.2 or later.
  • Publish TLS-RPT (_smtp._tls) for every mail-receiving domain and route reports to a parser.
  • Publish MTA-STS in testing, fix failures, then enforce with a stepped-up max_age.
  • Deploy DANE TLSA records (3 1 1) where your MX zones are DNSSEC-signed, with a rollover procedure tied to certificate renewal.
  • Enable MTA-STS and DANE validation on outbound MTAs, and mandatory verified TLS for named sensitive partners.
  • Offer implicit TLS on 465, require TLS before AUTH on 587, and disable cleartext IMAP and POP3.
  • Inventory and rotate SMTP AUTH credentials and relay API keys held by devices and applications.
  • Monitor the mta-sts host, its certificate, and changes to MX, TXT and TLSA records.
  • Document the removal procedure (mode: none, new id, wait max_age) in your mail migration runbook.

Standards and references

Related NHI Mgmt Group resources: DMARC, SPF and DKIM Guide · Email Security Guide · DNSSEC Deployment Guide · How TLS Works Guide · Exim and Postfix Mail Server Hardening Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.